[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:24:59 dist-test-slave-rmhg krb5kdc[15047](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:24:59 dist-test-slave-rmhg krb5kdc[15047](info): set up 2 sockets
Aug 10 23:24:59 dist-test-slave-rmhg krb5kdc[15047](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 23:25:00.820538 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.183s user 0.005s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:25:00 dist-test-slave-rmhg krb5kdc[15047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:25:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:25:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
I20260810 23:25:01.017787 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:44799
--webserver_interface=127.14.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:37183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:44799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:01.418066 15066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:01.418560 15066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:01.418704 15066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:01.427035 15066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:25:01.427212 15066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:01.427300 15066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:01.427407 15066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:25:01.427481 15066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:25:01.439538 15066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:37183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:44799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:44799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:01.442054 15066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:01.444684 15066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:01.469925 15073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:01.469786 15075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:01.469873 15072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:01.472229 15066 server_base.cc:1034] running on GCE node
I20260810 23:25:01.473963 15066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:01.476748 15066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:01.480163 15066 hybrid_clock.cc:648] HybridClock initialized: now 1786404301480014 us; error 178 us; skew 500 ppm
I20260810 23:25:01.480823 15066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:01 dist-test-slave-rmhg krb5kdc[15047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404301, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:01.485827 15066 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:25:01.488734 15066 webserver.cc:460] Webserver started at http://127.14.175.126:45137/ using document root <none> and password file <none>
I20260810 23:25:01.490113 15066 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:01.490283 15066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:01.490787 15066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:01.495738 15066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "3ebee54f54944094bab43091c1afa23e"
format_stamp: "Formatted at 2026-08-10 23:25:01 on dist-test-slave-rmhg"
server_key: "09ca80f86ed0e1f5af0fb176cd17e656"
server_key_iv: "274e925a425d1fbc1b01080e05e904ae"
server_key_version: "encryptionkey@0"
I20260810 23:25:01.496937 15066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "3ebee54f54944094bab43091c1afa23e"
format_stamp: "Formatted at 2026-08-10 23:25:01 on dist-test-slave-rmhg"
server_key: "09ca80f86ed0e1f5af0fb176cd17e656"
server_key_iv: "274e925a425d1fbc1b01080e05e904ae"
server_key_version: "encryptionkey@0"
I20260810 23:25:01.506654 15066 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.002s sys 0.005s
I20260810 23:25:01.514038 15083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:01.516520 15066 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260810 23:25:01.516901 15066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "3ebee54f54944094bab43091c1afa23e"
format_stamp: "Formatted at 2026-08-10 23:25:01 on dist-test-slave-rmhg"
server_key: "09ca80f86ed0e1f5af0fb176cd17e656"
server_key_iv: "274e925a425d1fbc1b01080e05e904ae"
server_key_version: "encryptionkey@0"
I20260810 23:25:01.517231 15066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:01.551096 15066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:01.556898 15066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:01.557319 15066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:01.582489 15066 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:44799
I20260810 23:25:01.582506 15164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:44799 every 8 connection(s)
I20260810 23:25:01.585444 15066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:25:01.591707 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 15066
I20260810 23:25:01.592234 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:25:01.593770 15165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:01.595187 15037 external_mini_cluster.cc:1428] Setting key 23e0aad244facbdf85259b5ce73dcc7c
I20260810 23:25:01.612810 15165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ebee54f54944094bab43091c1afa23e: Bootstrap starting.
I20260810 23:25:01.619496 15165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ebee54f54944094bab43091c1afa23e: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:01.621613 15165 log.cc:824] T 00000000000000000000000000000000 P 3ebee54f54944094bab43091c1afa23e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:25:01.627211 15165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ebee54f54944094bab43091c1afa23e: No bootstrap required, opened a new log
Aug 10 23:25:01 dist-test-slave-rmhg krb5kdc[15047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:01.639007 15165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ebee54f54944094bab43091c1afa23e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ebee54f54944094bab43091c1afa23e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 44799 } }
I20260810 23:25:01.639604 15165 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ebee54f54944094bab43091c1afa23e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:01.639809 15165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ebee54f54944094bab43091c1afa23e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ebee54f54944094bab43091c1afa23e, State: Initialized, Role: FOLLOWER
I20260810 23:25:01.641049 15165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ebee54f54944094bab43091c1afa23e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ebee54f54944094bab43091c1afa23e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 44799 } }
I20260810 23:25:01.641507 15165 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ebee54f54944094bab43091c1afa23e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:25:01.641886 15165 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ebee54f54944094bab43091c1afa23e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:25:01.642172 15165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ebee54f54944094bab43091c1afa23e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:01.646574 15172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:01.603266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34996 (local address 127.14.175.126:44799)
0810 23:25:01.604662 (+ 1396us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:01.604684 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:01.606064 (+ 1380us) server_negotiation.cc:407] Connection header received
0810 23:25:01.616485 (+ 10421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:01.616513 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:01.617236 (+ 723us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:01.617787 (+ 551us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:01.621765 (+ 3978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:01.623935 (+ 2170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:01.626919 (+ 2984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:01.627830 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:01.628127 (+ 297us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:01.635911 (+ 7784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:01.636001 (+ 90us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:01.636044 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:01.636106 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:01.639564 (+ 3458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:01.640682 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:01.640690 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:01.640696 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:01.640940 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:01.641554 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:01.641562 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:01.641565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:01.642667 (+ 1102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:01.643180 (+ 513us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:01.644226 (+ 1046us) server_negotiation.cc:299] Negotiation successful
0810 23:25:01.644769 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":915,"thread_start_us":438,"threads_started":1}
I20260810 23:25:01.699522 15165 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ebee54f54944094bab43091c1afa23e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ebee54f54944094bab43091c1afa23e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 44799 } }
I20260810 23:25:01.700893 15165 leader_election.cc:302] T 00000000000000000000000000000000 P 3ebee54f54944094bab43091c1afa23e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ebee54f54944094bab43091c1afa23e; no voters:
I20260810 23:25:01.702050 15165 leader_election.cc:288] T 00000000000000000000000000000000 P 3ebee54f54944094bab43091c1afa23e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:25:01.702685 15175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ebee54f54944094bab43091c1afa23e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:25:01.704165 15175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ebee54f54944094bab43091c1afa23e [term 1 LEADER]: Becoming Leader. State: Replica: 3ebee54f54944094bab43091c1afa23e, State: Running, Role: LEADER
I20260810 23:25:01.705257 15175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ebee54f54944094bab43091c1afa23e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ebee54f54944094bab43091c1afa23e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 44799 } }
I20260810 23:25:01.706310 15165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ebee54f54944094bab43091c1afa23e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:25:01.711229 15177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ebee54f54944094bab43091c1afa23e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ebee54f54944094bab43091c1afa23e. Latest consensus state: current_term: 1 leader_uuid: "3ebee54f54944094bab43091c1afa23e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ebee54f54944094bab43091c1afa23e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 44799 } } }
I20260810 23:25:01.711267 15176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ebee54f54944094bab43091c1afa23e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ebee54f54944094bab43091c1afa23e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ebee54f54944094bab43091c1afa23e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 44799 } } }
I20260810 23:25:01.711798 15177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ebee54f54944094bab43091c1afa23e [sys.catalog]: This master's current role is: LEADER
I20260810 23:25:01.711826 15176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ebee54f54944094bab43091c1afa23e [sys.catalog]: This master's current role is: LEADER
I20260810 23:25:01.716949 15187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:25:01.726387 15187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:25:01.744450 15187 catalog_manager.cc:1347] Generated new cluster ID: 1595ac619404428ebaa2862d2cdb012d
I20260810 23:25:01.744626 15187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:25:01.760489 15187 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:25:01.762822 15187 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:25:01.782145 15187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ebee54f54944094bab43091c1afa23e: Generated new TSK 0
I20260810 23:25:01.783764 15187 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
W20260810 23:25:02.609043 15037 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.14.175.65: real 0.814s user 0.002s sys 0.011s
I20260810 23:25:02.610112 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.65:0
--local_ip_for_outbound_sockets=127.14.175.65
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:44799
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:37183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:02.978860 15203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:02.979259 15203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:02.979406 15203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:02.987044 15203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:02.987179 15203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:02.987344 15203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.65
I20260810 23:25:02.997685 15203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:37183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:44799
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.65
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:02.999163 15203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:03.000627 15203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:03.016179 15218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:03.016230 15214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:03.016430 15212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:03.017074 15203 server_base.cc:1034] running on GCE node
I20260810 23:25:03.017753 15203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:03.018800 15203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:03.020074 15203 hybrid_clock.cc:648] HybridClock initialized: now 1786404303019975 us; error 59 us; skew 500 ppm
I20260810 23:25:03.020524 15203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:03 dist-test-slave-rmhg krb5kdc[15047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404303, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:03.024212 15203 init.cc:377] Logged in from keytab as kudu/127.14.175.65@KRBTEST.COM (short username kudu)
I20260810 23:25:03.025960 15203 webserver.cc:460] Webserver started at http://127.14.175.65:40183/ using document root <none> and password file <none>
I20260810 23:25:03.026700 15203 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:03.026844 15203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:03.027189 15203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:03.030459 15203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-0/data/instance:
uuid: "1f31492101c140c79a21376fcd121051"
format_stamp: "Formatted at 2026-08-10 23:25:03 on dist-test-slave-rmhg"
server_key: "f67c22ef098b256a197ec7be5a06641c"
server_key_iv: "8c1cdf25f5027a7493199e69b1f208de"
server_key_version: "encryptionkey@0"
I20260810 23:25:03.031313 15203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance:
uuid: "1f31492101c140c79a21376fcd121051"
format_stamp: "Formatted at 2026-08-10 23:25:03 on dist-test-slave-rmhg"
server_key: "f67c22ef098b256a197ec7be5a06641c"
server_key_iv: "8c1cdf25f5027a7493199e69b1f208de"
server_key_version: "encryptionkey@0"
I20260810 23:25:03.037281 15203 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260810 23:25:03.041344 15226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:03.042830 15203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:25:03.043025 15203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-0/wal
uuid: "1f31492101c140c79a21376fcd121051"
format_stamp: "Formatted at 2026-08-10 23:25:03 on dist-test-slave-rmhg"
server_key: "f67c22ef098b256a197ec7be5a06641c"
server_key_iv: "8c1cdf25f5027a7493199e69b1f208de"
server_key_version: "encryptionkey@0"
I20260810 23:25:03.043249 15203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:03.067412 15203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:03.072487 15203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:03.072849 15203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:03.074167 15203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:25:03.075992 15203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:25:03.076083 15203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:03.076275 15203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:25:03.076350 15203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:03.112766 15203 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.65:36553
I20260810 23:25:03.112860 15398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.65:36553 every 8 connection(s)
I20260810 23:25:03.114315 15203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
I20260810 23:25:03.124773 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 15203
I20260810 23:25:03.125020 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance
I20260810 23:25:03.125870 15037 external_mini_cluster.cc:1428] Setting key dc5608c523a10f403354ed94702c4e36
Aug 10 23:25:03 dist-test-slave-rmhg krb5kdc[15047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404303, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:03.139829 15401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:03.118040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:46255 (local address 127.14.175.126:44799)
0810 23:25:03.118613 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:03.118619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:03.119911 (+ 1292us) server_negotiation.cc:407] Connection header received
0810 23:25:03.122244 (+ 2333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:03.122249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:03.122334 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:03.122444 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:03.124495 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:03.125720 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:03.127638 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:03.128140 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:03.128287 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:03.132329 (+ 4042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:03.132336 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:03.132340 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:03.132386 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:03.135717 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:03.136921 (+ 1204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:03.136927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:03.136929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:03.137111 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:03.137678 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:03.137682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:03.137684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:03.137984 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:03.138147 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:03.139136 (+ 989us) server_negotiation.cc:299] Negotiation successful
0810 23:25:03.139402 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":245,"threads_started":1}
I20260810 23:25:03.141009 15402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:03.118759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:44799 (local address 127.14.175.65:46255)
0810 23:25:03.119705 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:03.119754 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:03.121816 (+ 2062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:03.122803 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:03.122813 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:03.123520 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:03.124217 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:03.124239 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:03.125907 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:03.125911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:03.127463 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:03.127479 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:03.128378 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:03.128383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:03.128657 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:03.129382 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:03.129407 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:03.132013 (+ 2606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:03.135931 (+ 3918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:03.135945 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:03.135968 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:03.136756 (+ 788us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:03.137281 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:03.137287 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:03.137291 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:03.137538 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:03.138147 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:03.138157 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:03.138606 (+ 449us) client_negotiation.cc:769] Sending connection context
0810 23:25:03.138984 (+ 378us) client_negotiation.cc:240] Negotiation successful
0810 23:25:03.139629 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":393,"threads_started":1}
I20260810 23:25:03.143050 15399 heartbeater.cc:347] Connected to a master server at 127.14.175.126:44799
I20260810 23:25:03.143561 15399 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.14.175.66@KRBTEST.COM; defaulting to no policy
I20260810 23:25:03.144738 15399 heartbeater.cc:511] Master 127.14.175.126:44799 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.66@KRBTEST.COM" created.
I20260810 23:25:03.149744 15113 ts_manager.cc:194] Registered new tserver with Master: 1f31492101c140c79a21376fcd121051 (127.14.175.65:36553)
I20260810 23:25:03.152787 15113 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.65@KRBTEST.COM'} at 127.14.175.65:46255
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
I20260810 23:25:03.172811 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.66:0
--local_ip_for_outbound_sockets=127.14.175.66
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:44799
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:37183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:03.709697 15405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:03.710184 15405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:03.710459 15405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:03.722518 15405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:03.722867 15405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:03.723147 15405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.66
I20260810 23:25:03.738870 15405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:37183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:44799
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.66
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:03.740677 15405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:03.742552 15405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:03.764955 15411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:03.765293 15413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:03.765518 15415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:03.767158 15405 server_base.cc:1034] running on GCE node
I20260810 23:25:03.768151 15405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:03.769615 15405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:03.770879 15405 hybrid_clock.cc:648] HybridClock initialized: now 1786404303770807 us; error 57 us; skew 500 ppm
I20260810 23:25:03.771359 15405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:03 dist-test-slave-rmhg krb5kdc[15047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404303, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:03.777107 15405 init.cc:377] Logged in from keytab as kudu/127.14.175.66@KRBTEST.COM (short username kudu)
I20260810 23:25:03.779516 15405 webserver.cc:460] Webserver started at http://127.14.175.66:42983/ using document root <none> and password file <none>
I20260810 23:25:03.780519 15405 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:03.780767 15405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:03.781424 15405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:03.785977 15405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-1/data/instance:
uuid: "e1bd059684e04ca4bfd2f9d113adae3c"
format_stamp: "Formatted at 2026-08-10 23:25:03 on dist-test-slave-rmhg"
server_key: "a7be8ee449f74478f4077913ff71e8a8"
server_key_iv: "07f519d872dc0c0b60bc2c4fd0eb4eb2"
server_key_version: "encryptionkey@0"
I20260810 23:25:03.787011 15405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance:
uuid: "e1bd059684e04ca4bfd2f9d113adae3c"
format_stamp: "Formatted at 2026-08-10 23:25:03 on dist-test-slave-rmhg"
server_key: "a7be8ee449f74478f4077913ff71e8a8"
server_key_iv: "07f519d872dc0c0b60bc2c4fd0eb4eb2"
server_key_version: "encryptionkey@0"
I20260810 23:25:03.795089 15405 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.002s
I20260810 23:25:03.801040 15428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:03.802757 15405 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:25:03.803130 15405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-1/wal
uuid: "e1bd059684e04ca4bfd2f9d113adae3c"
format_stamp: "Formatted at 2026-08-10 23:25:03 on dist-test-slave-rmhg"
server_key: "a7be8ee449f74478f4077913ff71e8a8"
server_key_iv: "07f519d872dc0c0b60bc2c4fd0eb4eb2"
server_key_version: "encryptionkey@0"
I20260810 23:25:03.803529 15405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:03.831499 15405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:03.837888 15405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:03.838430 15405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:03.840176 15405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:25:03.842511 15405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:25:03.842692 15405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:03.842921 15405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:25:03.843035 15405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:03.922016 15405 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.66:46013
I20260810 23:25:03.922102 15590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.66:46013 every 8 connection(s)
I20260810 23:25:03.923717 15405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
I20260810 23:25:03.928892 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 15405
I20260810 23:25:03.929207 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance
I20260810 23:25:03.929973 15037 external_mini_cluster.cc:1428] Setting key 8d94a4ce63dd6e52de2d5339d55bc282
Aug 10 23:25:03 dist-test-slave-rmhg krb5kdc[15047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404303, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.175.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.67@KRBTEST.COM" created.
I20260810 23:25:03.954344 15593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:03.927187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:39159 (local address 127.14.175.126:44799)
0810 23:25:03.927858 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:03.927864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:03.929685 (+ 1821us) server_negotiation.cc:407] Connection header received
0810 23:25:03.932556 (+ 2871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:03.932561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:03.932688 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:03.932782 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:03.935456 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:03.936404 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:03.939230 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:03.940258 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:03.940399 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:03.945504 (+ 5105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:03.945513 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:03.945518 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:03.945566 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:03.950079 (+ 4513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:03.951117 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:03.951125 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:03.951129 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:03.951401 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:03.952018 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:03.952022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:03.952024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:03.952402 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:03.952525 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:03.953704 (+ 1179us) server_negotiation.cc:299] Negotiation successful
0810 23:25:03.953970 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":404,"threads_started":1}
I20260810 23:25:03.956334 15595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:03.927898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:44799 (local address 127.14.175.66:39159)
0810 23:25:03.929457 (+ 1559us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:03.929527 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:03.932275 (+ 2748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:03.933128 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:03.933141 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:03.934012 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:03.935270 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:03.935301 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:03.936647 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:03.936652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:03.939056 (+ 2404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:03.939072 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:03.940497 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:03.940502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:03.940790 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:03.941558 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:03.941585 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:03.945098 (+ 3513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:03.950308 (+ 5210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:03.950318 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:03.950337 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:03.950951 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:03.951517 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:03.951520 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:03.951523 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:03.951897 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:03.952546 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:03.952556 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:03.953214 (+ 658us) client_negotiation.cc:769] Sending connection context
0810 23:25:03.953577 (+ 363us) client_negotiation.cc:240] Negotiation successful
0810 23:25:03.954137 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1193,"thread_start_us":783,"threads_started":1}
I20260810 23:25:03.958427 15591 heartbeater.cc:347] Connected to a master server at 127.14.175.126:44799
I20260810 23:25:03.958909 15591 heartbeater.cc:464] Registering TS with master...
I20260810 23:25:03.960263 15591 heartbeater.cc:511] Master 127.14.175.126:44799 requested a full tablet report, sending...
I20260810 23:25:03.962270 15113 ts_manager.cc:194] Registered new tserver with Master: e1bd059684e04ca4bfd2f9d113adae3c (127.14.175.66:46013)
I20260810 23:25:03.963385 15113 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.66@KRBTEST.COM'} at 127.14.175.66:39159
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
I20260810 23:25:03.980829 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.67:0
--local_ip_for_outbound_sockets=127.14.175.67
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:44799
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:37183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:25:04.157490 15399 heartbeater.cc:503] Master 127.14.175.126:44799 was elected leader, sending a full tablet report...
W20260810 23:25:04.343852 15601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:04.344195 15601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:04.344324 15601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:04.351879 15601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:04.352030 15601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:04.352196 15601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.67
I20260810 23:25:04.362424 15601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:37183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:44799
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.67
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:04.363837 15601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:04.365304 15601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:04.379990 15612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:04.380016 15618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:04.380318 15615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:04.381527 15601 server_base.cc:1034] running on GCE node
I20260810 23:25:04.382330 15601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:04.383809 15601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:04.387176 15601 hybrid_clock.cc:648] HybridClock initialized: now 1786404304387093 us; error 83 us; skew 500 ppm
I20260810 23:25:04.387586 15601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:04 dist-test-slave-rmhg krb5kdc[15047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404304, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:04.390987 15601 init.cc:377] Logged in from keytab as kudu/127.14.175.67@KRBTEST.COM (short username kudu)
I20260810 23:25:04.392503 15601 webserver.cc:460] Webserver started at http://127.14.175.67:42331/ using document root <none> and password file <none>
I20260810 23:25:04.393226 15601 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:04.393339 15601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:04.393633 15601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:04.396623 15601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-2/data/instance:
uuid: "6f11d8c6223b4cfd9aed11010cc41053"
format_stamp: "Formatted at 2026-08-10 23:25:04 on dist-test-slave-rmhg"
server_key: "128985d867bee9b40d0a51f0b7e449d1"
server_key_iv: "d2e92961165c03115afdcfd3e1fc6742"
server_key_version: "encryptionkey@0"
I20260810 23:25:04.397418 15601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance:
uuid: "6f11d8c6223b4cfd9aed11010cc41053"
format_stamp: "Formatted at 2026-08-10 23:25:04 on dist-test-slave-rmhg"
server_key: "128985d867bee9b40d0a51f0b7e449d1"
server_key_iv: "d2e92961165c03115afdcfd3e1fc6742"
server_key_version: "encryptionkey@0"
I20260810 23:25:04.402827 15601 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 23:25:04.406399 15625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:04.407815 15601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:25:04.407974 15601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-2/wal
uuid: "6f11d8c6223b4cfd9aed11010cc41053"
format_stamp: "Formatted at 2026-08-10 23:25:04 on dist-test-slave-rmhg"
server_key: "128985d867bee9b40d0a51f0b7e449d1"
server_key_iv: "d2e92961165c03115afdcfd3e1fc6742"
server_key_version: "encryptionkey@0"
I20260810 23:25:04.408159 15601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:04.422501 15601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:04.426633 15601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:04.426937 15601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:04.428020 15601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:25:04.429512 15601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:25:04.429590 15601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:04.429687 15601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:25:04.429734 15601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:04.462852 15601 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.67:34013
I20260810 23:25:04.462913 15797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.67:34013 every 8 connection(s)
I20260810 23:25:04.464274 15601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
I20260810 23:25:04.471051 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 15601
I20260810 23:25:04.471217 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance
I20260810 23:25:04.471689 15037 external_mini_cluster.cc:1428] Setting key 38a3aff24d94c39e27207bda9dce63fb
Aug 10 23:25:04 dist-test-slave-rmhg krb5kdc[15047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404304, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:04.486361 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:04.467144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:34101 (local address 127.14.175.126:44799)
0810 23:25:04.467723 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:04.467728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:04.468993 (+ 1265us) server_negotiation.cc:407] Connection header received
0810 23:25:04.471063 (+ 2070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:04.471069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:04.471185 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:04.471302 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:04.473221 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:04.474124 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:04.475961 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:04.476381 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:04.476497 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:04.480165 (+ 3668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:04.480171 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:04.480174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:04.480211 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:04.482951 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:04.483787 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:04.483792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:04.483794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:04.483928 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:04.484383 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:04.484386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:04.484389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:04.484707 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:04.484855 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:04.485714 (+ 859us) server_negotiation.cc:299] Negotiation successful
0810 23:25:04.485960 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":249,"threads_started":1}
I20260810 23:25:04.487445 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:04.467770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:44799 (local address 127.14.175.67:34101)
0810 23:25:04.468776 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:04.468829 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:04.470724 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:04.471620 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:04.471630 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:04.472238 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:04.472987 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:04.473018 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:04.474307 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:04.474313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:04.475821 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:04.475834 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:04.476629 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:04.476638 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:04.476872 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:04.477578 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:04.477603 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:04.479901 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:04.483144 (+ 3243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:04.483156 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:04.483178 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:04.483650 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:04.484042 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:04.484047 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:04.484060 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:04.484266 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:04.484841 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:04.484850 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:04.485302 (+ 452us) client_negotiation.cc:769] Sending connection context
0810 23:25:04.485588 (+ 286us) client_negotiation.cc:240] Negotiation successful
0810 23:25:04.486190 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":379,"threads_started":1}
I20260810 23:25:04.489084 15798 heartbeater.cc:347] Connected to a master server at 127.14.175.126:44799
I20260810 23:25:04.489431 15798 heartbeater.cc:464] Registering TS with master...
I20260810 23:25:04.490299 15798 heartbeater.cc:511] Master 127.14.175.126:44799 requested a full tablet report, sending...
I20260810 23:25:04.491796 15113 ts_manager.cc:194] Registered new tserver with Master: 6f11d8c6223b4cfd9aed11010cc41053 (127.14.175.67:34013)
I20260810 23:25:04.492640 15113 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.67@KRBTEST.COM'} at 127.14.175.67:34101
I20260810 23:25:04.502000 15037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 23:25:04 dist-test-slave-rmhg krb5kdc[15047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404304, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 23:25:04 dist-test-slave-rmhg krb5kdc[15047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404304, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.175.65@KRBTEST.COM
I20260810 23:25:04.552824 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:04.536301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51054 (local address 127.14.175.65:36553)
0810 23:25:04.537068 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:04.537077 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:04.537113 (+ 36us) server_negotiation.cc:407] Connection header received
0810 23:25:04.537237 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:04.537246 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:04.537468 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:04.537645 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:04.539134 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:04.540182 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:04.541668 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:04.542284 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:04.542412 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:04.546669 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:04.546712 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:04.546730 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:04.546774 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:04.550037 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:04.550879 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:04.550887 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:04.550890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:04.551029 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:04.551383 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:04.551386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:04.551388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:04.551771 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:04.552009 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:04.552304 (+ 295us) server_negotiation.cc:299] Negotiation successful
0810 23:25:04.552516 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":380,"threads_started":1}
W20260810 23:25:04.554387 15264 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51054
Aug 10 23:25:04 dist-test-slave-rmhg krb5kdc[15047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 23:25:04 dist-test-slave-rmhg krb5kdc[15047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.65@KRBTEST.COM
I20260810 23:25:04.606581 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:04.592087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51058 (local address 127.14.175.65:36553)
0810 23:25:04.592331 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:04.592339 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:04.592981 (+ 642us) server_negotiation.cc:407] Connection header received
0810 23:25:04.593197 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:04.593203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:04.593312 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:04.593402 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:04.594909 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:04.595668 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:04.596985 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:04.597471 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:04.597580 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:04.601547 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:04.601553 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:04.601557 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:04.601598 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:04.604224 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:04.604839 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:04.604843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:04.604845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:04.604979 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:04.605366 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:04.605369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:04.605371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:04.605649 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:04.605830 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:04.606059 (+ 229us) server_negotiation.cc:299] Negotiation successful
0810 23:25:04.606258 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:25:04.608628 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 15203
I20260810 23:25:04.622747 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:04.609988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51064 (local address 127.14.175.65:36553)
0810 23:25:04.610220 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:04.610226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:04.610490 (+ 264us) server_negotiation.cc:407] Connection header received
0810 23:25:04.610620 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:04.610624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:04.610685 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:04.610770 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:04.611965 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:04.612713 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:04.614120 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:04.614559 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:04.614659 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:04.616911 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:04.616917 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:04.616920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:04.616957 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:04.619761 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:04.620630 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:04.620637 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:04.620641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:04.620801 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:04.621275 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:04.621279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:04.621281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:04.621599 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:04.621794 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:04.622203 (+ 409us) server_negotiation.cc:299] Negotiation successful
0810 23:25:04.622412 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 23:25:04.623718 15391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:25:04.714198 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 15203
I20260810 23:25:04.735891 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 15405
Aug 10 23:25:04 dist-test-slave-rmhg krb5kdc[15047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.66@KRBTEST.COM
I20260810 23:25:04.755479 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:04.736805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43408 (local address 127.14.175.66:46013)
0810 23:25:04.737455 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:04.737464 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:04.737499 (+ 35us) server_negotiation.cc:407] Connection header received
0810 23:25:04.737570 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:04.737577 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:04.737819 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:04.737974 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:04.739210 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:04.740233 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:04.741689 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:04.742272 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:04.742376 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:04.749736 (+ 7360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:04.749753 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:04.749771 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:04.749811 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:04.752641 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:04.753427 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:04.753436 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:04.753438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:04.753579 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:04.753978 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:04.753981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:04.753984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:04.754336 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:04.754578 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:04.754839 (+ 261us) server_negotiation.cc:299] Negotiation successful
0810 23:25:04.755054 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":364,"threads_started":1}
I20260810 23:25:04.756830 15584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:25:04.838847 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 15405
I20260810 23:25:04.865417 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 15601
Aug 10 23:25:04 dist-test-slave-rmhg krb5kdc[15047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.67@KRBTEST.COM
I20260810 23:25:04.882284 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:04.866349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36894 (local address 127.14.175.67:34013)
0810 23:25:04.866984 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:04.866991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:04.867023 (+ 32us) server_negotiation.cc:407] Connection header received
0810 23:25:04.867094 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:04.867101 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:04.867329 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:04.867462 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:04.868706 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:04.869858 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:04.871141 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:04.871750 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:04.871854 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:04.876267 (+ 4413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:04.876285 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:04.876305 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:04.876351 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:04.879247 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:04.880001 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:04.880009 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:04.880011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:04.880157 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:04.880664 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:04.880668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:04.880671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:04.881088 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:04.881406 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:04.881713 (+ 307us) server_negotiation.cc:299] Negotiation successful
0810 23:25:04.881958 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":303,"threads_started":1}
I20260810 23:25:04.883888 15793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:25:04.967396 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 15601
I20260810 23:25:04.989832 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 15066
I20260810 23:25:04.991040 15159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:25:05.098497 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 15066
2026-08-10T23:25:05Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5527 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:25:05 dist-test-slave-rmhg krb5kdc[15828](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:25:05 dist-test-slave-rmhg krb5kdc[15828](info): set up 2 sockets
Aug 10 23:25:05 dist-test-slave-rmhg krb5kdc[15828](info): commencing operation
krb5kdc: starting...
W20260810 23:25:06.312655 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:25:06 dist-test-slave-rmhg krb5kdc[15828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:25:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:25:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
I20260810 23:25:06.521721 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:34363
--webserver_interface=127.14.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:37389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:34363
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:06.876791 15849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:06.877182 15849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:06.877329 15849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:06.885090 15849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:25:06.885277 15849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:06.885365 15849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:06.885444 15849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:25:06.885510 15849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:25:06.896018 15849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:37389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:34363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:34363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:06.897514 15849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:06.898945 15849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:06.911180 15856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:06.911142 15857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:06.911142 15860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:06.912058 15849 server_base.cc:1034] running on GCE node
I20260810 23:25:06.912914 15849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:06.914501 15849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:06.915723 15849 hybrid_clock.cc:648] HybridClock initialized: now 1786404306915659 us; error 57 us; skew 500 ppm
I20260810 23:25:06.916086 15849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:06 dist-test-slave-rmhg krb5kdc[15828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404306, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:06.919652 15849 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:25:06.921456 15849 webserver.cc:460] Webserver started at http://127.14.175.126:42843/ using document root <none> and password file <none>
I20260810 23:25:06.922252 15849 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:06.922386 15849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:06.922725 15849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:06.926011 15849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "29ed302e2b714fc781d1a98b60ab8daf"
format_stamp: "Formatted at 2026-08-10 23:25:06 on dist-test-slave-rmhg"
server_key: "f1f9b142034537af1caf9aad24800fe0"
server_key_iv: "a29ad1ec0e5604259e2159899d80d0d8"
server_key_version: "encryptionkey@0"
I20260810 23:25:06.926820 15849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "29ed302e2b714fc781d1a98b60ab8daf"
format_stamp: "Formatted at 2026-08-10 23:25:06 on dist-test-slave-rmhg"
server_key: "f1f9b142034537af1caf9aad24800fe0"
server_key_iv: "a29ad1ec0e5604259e2159899d80d0d8"
server_key_version: "encryptionkey@0"
I20260810 23:25:06.932602 15849 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 23:25:06.936718 15871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:06.938333 15849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:25:06.938544 15849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "29ed302e2b714fc781d1a98b60ab8daf"
format_stamp: "Formatted at 2026-08-10 23:25:06 on dist-test-slave-rmhg"
server_key: "f1f9b142034537af1caf9aad24800fe0"
server_key_iv: "a29ad1ec0e5604259e2159899d80d0d8"
server_key_version: "encryptionkey@0"
I20260810 23:25:06.938797 15849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:06.965842 15849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:06.970566 15849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:06.970870 15849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:06.990159 15849 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:34363
I20260810 23:25:06.990201 15955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:34363 every 8 connection(s)
I20260810 23:25:06.991695 15849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:25:06.993724 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 15849
I20260810 23:25:06.993950 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:25:06.994745 15037 external_mini_cluster.cc:1428] Setting key dbd39b68296f1d853685b0870eaa25ca
I20260810 23:25:06.996896 15959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 23:25:07 dist-test-slave-rmhg krb5kdc[15828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:07.011632 15959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29ed302e2b714fc781d1a98b60ab8daf: Bootstrap starting.
I20260810 23:25:07.015226 15959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29ed302e2b714fc781d1a98b60ab8daf: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:07.016598 15959 log.cc:824] T 00000000000000000000000000000000 P 29ed302e2b714fc781d1a98b60ab8daf: Log is configured to *not* fsync() on all Append() calls
I20260810 23:25:07.019568 15965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:06.996951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58648 (local address 127.14.175.126:34363)
0810 23:25:06.997812 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:06.997824 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:06.997871 (+ 47us) server_negotiation.cc:407] Connection header received
0810 23:25:06.999531 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:06.999557 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:07.000006 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:07.000407 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:07.002457 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:07.003909 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:07.005742 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:07.006494 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:07.006639 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:07.011738 (+ 5099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:07.011749 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:07.011769 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:07.011827 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:07.014961 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:07.015906 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:07.015914 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:07.015921 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:07.016111 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:07.016550 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:07.016553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:07.016555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:07.017293 (+ 738us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:07.017632 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:07.018092 (+ 460us) server_negotiation.cc:299] Negotiation successful
0810 23:25:07.018523 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":394,"threads_started":1}
I20260810 23:25:07.019964 15959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29ed302e2b714fc781d1a98b60ab8daf: No bootstrap required, opened a new log
I20260810 23:25:07.027700 15959 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29ed302e2b714fc781d1a98b60ab8daf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ed302e2b714fc781d1a98b60ab8daf" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34363 } }
I20260810 23:25:07.028106 15959 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29ed302e2b714fc781d1a98b60ab8daf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:07.028218 15959 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29ed302e2b714fc781d1a98b60ab8daf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29ed302e2b714fc781d1a98b60ab8daf, State: Initialized, Role: FOLLOWER
I20260810 23:25:07.028923 15959 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29ed302e2b714fc781d1a98b60ab8daf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ed302e2b714fc781d1a98b60ab8daf" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34363 } }
I20260810 23:25:07.029209 15959 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29ed302e2b714fc781d1a98b60ab8daf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:25:07.029335 15959 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29ed302e2b714fc781d1a98b60ab8daf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:25:07.029472 15959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29ed302e2b714fc781d1a98b60ab8daf [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:07.031708 15959 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29ed302e2b714fc781d1a98b60ab8daf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ed302e2b714fc781d1a98b60ab8daf" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34363 } }
I20260810 23:25:07.032246 15959 leader_election.cc:302] T 00000000000000000000000000000000 P 29ed302e2b714fc781d1a98b60ab8daf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29ed302e2b714fc781d1a98b60ab8daf; no voters:
I20260810 23:25:07.032992 15959 leader_election.cc:288] T 00000000000000000000000000000000 P 29ed302e2b714fc781d1a98b60ab8daf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:25:07.033124 15970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29ed302e2b714fc781d1a98b60ab8daf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:25:07.035718 15959 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29ed302e2b714fc781d1a98b60ab8daf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:25:07.035650 15970 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29ed302e2b714fc781d1a98b60ab8daf [term 1 LEADER]: Becoming Leader. State: Replica: 29ed302e2b714fc781d1a98b60ab8daf, State: Running, Role: LEADER
I20260810 23:25:07.036245 15970 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29ed302e2b714fc781d1a98b60ab8daf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ed302e2b714fc781d1a98b60ab8daf" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34363 } }
I20260810 23:25:07.039237 15971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29ed302e2b714fc781d1a98b60ab8daf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29ed302e2b714fc781d1a98b60ab8daf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ed302e2b714fc781d1a98b60ab8daf" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34363 } } }
I20260810 23:25:07.039695 15971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29ed302e2b714fc781d1a98b60ab8daf [sys.catalog]: This master's current role is: LEADER
I20260810 23:25:07.040583 15972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29ed302e2b714fc781d1a98b60ab8daf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29ed302e2b714fc781d1a98b60ab8daf. Latest consensus state: current_term: 1 leader_uuid: "29ed302e2b714fc781d1a98b60ab8daf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ed302e2b714fc781d1a98b60ab8daf" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34363 } } }
I20260810 23:25:07.040916 15972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29ed302e2b714fc781d1a98b60ab8daf [sys.catalog]: This master's current role is: LEADER
I20260810 23:25:07.042470 15988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:25:07.048137 15988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:25:07.056869 15988 catalog_manager.cc:1347] Generated new cluster ID: 9ddc8a40c156419280f5b55b85ad2734
I20260810 23:25:07.056996 15988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:25:07.083024 15988 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:25:07.084587 15988 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:25:07.092737 15988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29ed302e2b714fc781d1a98b60ab8daf: Generated new TSK 0
I20260810 23:25:07.093722 15988 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
I20260810 23:25:07.153448 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.65:0
--local_ip_for_outbound_sockets=127.14.175.65
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:34363
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:37389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:07.517370 15997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:07.517689 15997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:07.517796 15997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:07.525607 15997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:07.525739 15997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:07.525861 15997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.65
I20260810 23:25:07.536453 15997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:37389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:34363
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.65
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:07.537963 15997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:07.539477 15997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:07.553942 16003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:07.553967 16011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:07.553996 16007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:07.554939 15997 server_base.cc:1034] running on GCE node
I20260810 23:25:07.555653 15997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:07.556722 15997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:07.557930 15997 hybrid_clock.cc:648] HybridClock initialized: now 1786404307557877 us; error 49 us; skew 500 ppm
I20260810 23:25:07.558274 15997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:07 dist-test-slave-rmhg krb5kdc[15828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404307, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:07.561548 15997 init.cc:377] Logged in from keytab as kudu/127.14.175.65@KRBTEST.COM (short username kudu)
I20260810 23:25:07.563113 15997 webserver.cc:460] Webserver started at http://127.14.175.65:38759/ using document root <none> and password file <none>
I20260810 23:25:07.563833 15997 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:07.563958 15997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:07.564287 15997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:07.567410 15997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-0/data/instance:
uuid: "98911e534274415d832c6f424a7f7fdc"
format_stamp: "Formatted at 2026-08-10 23:25:07 on dist-test-slave-rmhg"
server_key: "2d33686dafae2569ff96453c28150f03"
server_key_iv: "96dfc36034e55df98ae66fc8d58bcb98"
server_key_version: "encryptionkey@0"
I20260810 23:25:07.568218 15997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance:
uuid: "98911e534274415d832c6f424a7f7fdc"
format_stamp: "Formatted at 2026-08-10 23:25:07 on dist-test-slave-rmhg"
server_key: "2d33686dafae2569ff96453c28150f03"
server_key_iv: "96dfc36034e55df98ae66fc8d58bcb98"
server_key_version: "encryptionkey@0"
I20260810 23:25:07.573603 15997 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 23:25:07.577397 16021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:07.578928 15997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:25:07.579100 15997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-0/wal
uuid: "98911e534274415d832c6f424a7f7fdc"
format_stamp: "Formatted at 2026-08-10 23:25:07 on dist-test-slave-rmhg"
server_key: "2d33686dafae2569ff96453c28150f03"
server_key_iv: "96dfc36034e55df98ae66fc8d58bcb98"
server_key_version: "encryptionkey@0"
I20260810 23:25:07.579313 15997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:07.612393 15997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:07.616741 15997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:07.617033 15997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:07.618095 15997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:25:07.619539 15997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:25:07.619627 15997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:07.619763 15997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:25:07.619820 15997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:07.652349 15997 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.65:46701
I20260810 23:25:07.652396 16209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.65:46701 every 8 connection(s)
I20260810 23:25:07.653730 15997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
I20260810 23:25:07.657742 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 15997
I20260810 23:25:07.657958 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance
I20260810 23:25:07.658793 15037 external_mini_cluster.cc:1428] Setting key 0719424785840f43d5bc6f16023f2529
Aug 10 23:25:07 dist-test-slave-rmhg krb5kdc[15828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404307, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:07.676939 16214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:07.656412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:54459 (local address 127.14.175.126:34363)
0810 23:25:07.656904 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:07.656911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:07.659488 (+ 2577us) server_negotiation.cc:407] Connection header received
0810 23:25:07.660522 (+ 1034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:07.660528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:07.660623 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:07.660718 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:07.663051 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:07.663808 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:07.666103 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:07.666622 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:07.666746 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:07.670740 (+ 3994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:07.670746 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:07.670749 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:07.670789 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:07.673343 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:07.674364 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:07.674369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:07.674371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:07.674503 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:07.675046 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:07.675050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:07.675052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:07.675346 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:07.675464 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:07.676464 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0810 23:25:07.676660 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":230,"threads_started":1}
WARNING: no policy specified for kudu/127.14.175.66@KRBTEST.COM; defaulting to no policy
I20260810 23:25:07.678551 16216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:07.657003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:34363 (local address 127.14.175.65:54459)
0810 23:25:07.657887 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:07.657963 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:07.660240 (+ 2277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:07.661029 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:07.661042 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:07.661892 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:07.662803 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:07.662834 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:07.664001 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:07.664007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:07.665944 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:07.665963 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:07.666884 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:07.666891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:07.667199 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:07.667879 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:07.667908 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:07.670439 (+ 2531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:07.673509 (+ 3070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:07.673519 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:07.673542 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:07.674231 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:07.674614 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:07.674619 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:07.674623 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:07.674917 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:07.675483 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:07.675493 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:07.676006 (+ 513us) client_negotiation.cc:769] Sending connection context
0810 23:25:07.676340 (+ 334us) client_negotiation.cc:240] Negotiation successful
0810 23:25:07.676965 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":342,"threads_started":1}
I20260810 23:25:07.680406 16211 heartbeater.cc:347] Connected to a master server at 127.14.175.126:34363
I20260810 23:25:07.680720 16211 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.66@KRBTEST.COM" created.
I20260810 23:25:07.681716 16211 heartbeater.cc:511] Master 127.14.175.126:34363 requested a full tablet report, sending...
I20260810 23:25:07.684237 15901 ts_manager.cc:194] Registered new tserver with Master: 98911e534274415d832c6f424a7f7fdc (127.14.175.65:46701)
I20260810 23:25:07.685828 15901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.65@KRBTEST.COM'} at 127.14.175.65:54459
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
I20260810 23:25:07.705533 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.66:0
--local_ip_for_outbound_sockets=127.14.175.66
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:34363
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:37389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:08.043984 16220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:08.044333 16220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:08.044494 16220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:08.052114 16220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:08.052263 16220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:08.052426 16220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.66
I20260810 23:25:08.062660 16220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:37389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:34363
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.66
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:08.064067 16220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:08.065503 16220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:08.078989 16227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:08.078969 16228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:08.078924 16232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:08.079985 16220 server_base.cc:1034] running on GCE node
I20260810 23:25:08.080654 16220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:08.081761 16220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:08.083050 16220 hybrid_clock.cc:648] HybridClock initialized: now 1786404308082966 us; error 50 us; skew 500 ppm
I20260810 23:25:08.083392 16220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:08 dist-test-slave-rmhg krb5kdc[15828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404308, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:08.086848 16220 init.cc:377] Logged in from keytab as kudu/127.14.175.66@KRBTEST.COM (short username kudu)
I20260810 23:25:08.088446 16220 webserver.cc:460] Webserver started at http://127.14.175.66:33563/ using document root <none> and password file <none>
I20260810 23:25:08.089195 16220 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:08.089329 16220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:08.089663 16220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:08.092748 16220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-1/data/instance:
uuid: "c5527e95bfdd49a98cab8c4661998c2b"
format_stamp: "Formatted at 2026-08-10 23:25:08 on dist-test-slave-rmhg"
server_key: "429afba0a280b964f9bd3be3da000761"
server_key_iv: "141b0efd7b7c4f508833cdc7d69dd578"
server_key_version: "encryptionkey@0"
I20260810 23:25:08.093708 16220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance:
uuid: "c5527e95bfdd49a98cab8c4661998c2b"
format_stamp: "Formatted at 2026-08-10 23:25:08 on dist-test-slave-rmhg"
server_key: "429afba0a280b964f9bd3be3da000761"
server_key_iv: "141b0efd7b7c4f508833cdc7d69dd578"
server_key_version: "encryptionkey@0"
I20260810 23:25:08.099203 16220 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:25:08.103060 16246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:08.104730 16220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:25:08.104921 16220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-1/wal
uuid: "c5527e95bfdd49a98cab8c4661998c2b"
format_stamp: "Formatted at 2026-08-10 23:25:08 on dist-test-slave-rmhg"
server_key: "429afba0a280b964f9bd3be3da000761"
server_key_iv: "141b0efd7b7c4f508833cdc7d69dd578"
server_key_version: "encryptionkey@0"
I20260810 23:25:08.105141 16220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:08.129643 16220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:08.134684 16220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:08.134999 16220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:08.136139 16220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:25:08.137717 16220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:25:08.137806 16220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:08.137913 16220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:25:08.137964 16220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:08.170143 16220 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.66:38199
I20260810 23:25:08.170207 16429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.66:38199 every 8 connection(s)
I20260810 23:25:08.171526 16220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
I20260810 23:25:08.175912 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 16220
I20260810 23:25:08.176105 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance
I20260810 23:25:08.176688 15037 external_mini_cluster.cc:1428] Setting key 68b0d18a88aa934ed39711c9f02a2d4b
Aug 10 23:25:08 dist-test-slave-rmhg krb5kdc[15828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404308, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:08.194653 16214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:08.174285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:34975 (local address 127.14.175.126:34363)
0810 23:25:08.174477 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:08.174482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:08.175854 (+ 1372us) server_negotiation.cc:407] Connection header received
0810 23:25:08.178232 (+ 2378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:08.178238 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:08.178328 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:08.178438 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:08.180768 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:08.181643 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:08.184069 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:08.184597 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:08.184722 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:08.188513 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:08.188519 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:08.188522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:08.188562 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:08.191300 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:08.192124 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:08.192128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:08.192130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:08.192271 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:08.192735 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:08.192739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:08.192741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:08.193032 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:08.193205 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:08.194033 (+ 828us) server_negotiation.cc:299] Negotiation successful
0810 23:25:08.194298 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
WARNING: no policy specified for kudu/127.14.175.67@KRBTEST.COM; defaulting to no policy
I20260810 23:25:08.195779 16432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:08.174796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:34363 (local address 127.14.175.66:34975)
0810 23:25:08.175632 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:08.175692 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:08.177929 (+ 2237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:08.178725 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:08.178734 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:08.179556 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:08.180481 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:08.180513 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:08.181842 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:08.181848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:08.183856 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:08.183874 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:08.184867 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:08.184874 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:08.185237 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:08.185945 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:08.185970 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:08.188236 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:08.191471 (+ 3235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:08.191481 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:08.191502 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:08.191983 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:08.192387 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:08.192392 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:08.192397 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:08.192615 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:08.193197 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:08.193207 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:08.193595 (+ 388us) client_negotiation.cc:769] Sending connection context
0810 23:25:08.193915 (+ 320us) client_negotiation.cc:240] Negotiation successful
0810 23:25:08.194524 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":292,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.67@KRBTEST.COM" created.
I20260810 23:25:08.197417 16430 heartbeater.cc:347] Connected to a master server at 127.14.175.126:34363
I20260810 23:25:08.197786 16430 heartbeater.cc:464] Registering TS with master...
I20260810 23:25:08.198583 16430 heartbeater.cc:511] Master 127.14.175.126:34363 requested a full tablet report, sending...
I20260810 23:25:08.200281 15901 ts_manager.cc:194] Registered new tserver with Master: c5527e95bfdd49a98cab8c4661998c2b (127.14.175.66:38199)
I20260810 23:25:08.201129 15901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.66@KRBTEST.COM'} at 127.14.175.66:34975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
I20260810 23:25:08.221311 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.67:0
--local_ip_for_outbound_sockets=127.14.175.67
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:34363
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:37389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:08.559713 16436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:08.560017 16436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:08.560117 16436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:08.567617 16436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:08.567762 16436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:08.567886 16436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.67
I20260810 23:25:08.578171 16436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:37389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:34363
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.67
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:08.579560 16436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:08.580926 16436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:08.594672 16443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:08.594789 16442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:08.594712 16447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:08.595459 16436 server_base.cc:1034] running on GCE node
I20260810 23:25:08.596095 16436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:08.597114 16436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:08.598366 16436 hybrid_clock.cc:648] HybridClock initialized: now 1786404308598312 us; error 36 us; skew 500 ppm
I20260810 23:25:08.598708 16436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:08 dist-test-slave-rmhg krb5kdc[15828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404308, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:08.602129 16436 init.cc:377] Logged in from keytab as kudu/127.14.175.67@KRBTEST.COM (short username kudu)
I20260810 23:25:08.603721 16436 webserver.cc:460] Webserver started at http://127.14.175.67:33795/ using document root <none> and password file <none>
I20260810 23:25:08.604480 16436 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:08.604619 16436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:08.604954 16436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:08.608143 16436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-2/data/instance:
uuid: "908a43fd4c954206aa8900fe71b0ac36"
format_stamp: "Formatted at 2026-08-10 23:25:08 on dist-test-slave-rmhg"
server_key: "39d7435eb1da29b22454c77ec9039ec2"
server_key_iv: "38c8bc4e006f0729cbe36fb1d0984de3"
server_key_version: "encryptionkey@0"
I20260810 23:25:08.608996 16436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance:
uuid: "908a43fd4c954206aa8900fe71b0ac36"
format_stamp: "Formatted at 2026-08-10 23:25:08 on dist-test-slave-rmhg"
server_key: "39d7435eb1da29b22454c77ec9039ec2"
server_key_iv: "38c8bc4e006f0729cbe36fb1d0984de3"
server_key_version: "encryptionkey@0"
I20260810 23:25:08.614717 16436 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 23:25:08.618522 16457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:08.620048 16436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260810 23:25:08.620241 16436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-2/wal
uuid: "908a43fd4c954206aa8900fe71b0ac36"
format_stamp: "Formatted at 2026-08-10 23:25:08 on dist-test-slave-rmhg"
server_key: "39d7435eb1da29b22454c77ec9039ec2"
server_key_iv: "38c8bc4e006f0729cbe36fb1d0984de3"
server_key_version: "encryptionkey@0"
I20260810 23:25:08.620448 16436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:08.638163 16436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:08.642678 16436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:08.643044 16436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:08.644138 16436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:25:08.645731 16436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:25:08.645812 16436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:08.645939 16436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:25:08.645994 16436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:08.679358 16436 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.67:46163
I20260810 23:25:08.679423 16637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.67:46163 every 8 connection(s)
I20260810 23:25:08.680728 16436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
I20260810 23:25:08.681207 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 16436
I20260810 23:25:08.681404 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance
I20260810 23:25:08.681968 15037 external_mini_cluster.cc:1428] Setting key 13fd69749bf003980e7eed54e329b4e8
I20260810 23:25:08.688961 16211 heartbeater.cc:503] Master 127.14.175.126:34363 was elected leader, sending a full tablet report...
Aug 10 23:25:08 dist-test-slave-rmhg krb5kdc[15828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404308, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:08.703691 16214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:08.683890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:43143 (local address 127.14.175.126:34363)
0810 23:25:08.684087 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:08.684092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:08.685798 (+ 1706us) server_negotiation.cc:407] Connection header received
0810 23:25:08.687998 (+ 2200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:08.688004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:08.688099 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:08.688194 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:08.690143 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:08.690921 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:08.692577 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:08.692999 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:08.693120 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:08.697210 (+ 4090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:08.697217 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:08.697223 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:08.697269 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:08.700415 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:08.701338 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:08.701343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:08.701346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:08.701490 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:08.701931 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:08.701934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:08.701936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:08.702227 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:08.702351 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:08.703196 (+ 845us) server_negotiation.cc:299] Negotiation successful
0810 23:25:08.703407 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 23:25:08.704851 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:08.684675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:34363 (local address 127.14.175.67:43143)
0810 23:25:08.685636 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:08.685691 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:08.687633 (+ 1942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:08.688493 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:08.688503 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:08.689144 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:08.689964 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:08.689987 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:08.691101 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:08.691105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:08.692421 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:08.692436 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:08.693280 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:08.693289 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:08.693531 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:08.694308 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:08.694334 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:08.696912 (+ 2578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:08.700597 (+ 3685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:08.700608 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:08.700625 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:08.701217 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:08.701594 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:08.701597 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:08.701599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:08.701835 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:08.702354 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:08.702361 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:08.702791 (+ 430us) client_negotiation.cc:769] Sending connection context
0810 23:25:08.703057 (+ 266us) client_negotiation.cc:240] Negotiation successful
0810 23:25:08.703572 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":427,"threads_started":1}
I20260810 23:25:08.706607 16639 heartbeater.cc:347] Connected to a master server at 127.14.175.126:34363
I20260810 23:25:08.706941 16639 heartbeater.cc:464] Registering TS with master...
I20260810 23:25:08.707926 16639 heartbeater.cc:511] Master 127.14.175.126:34363 requested a full tablet report, sending...
I20260810 23:25:08.709880 15901 ts_manager.cc:194] Registered new tserver with Master: 908a43fd4c954206aa8900fe71b0ac36 (127.14.175.67:46163)
I20260810 23:25:08.710951 15901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.67@KRBTEST.COM'} at 127.14.175.67:43143
I20260810 23:25:08.711798 15037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:25:08.734164 16214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:08.720410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58650 (local address 127.14.175.126:34363)
0810 23:25:08.720614 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:08.720621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:08.721142 (+ 521us) server_negotiation.cc:407] Connection header received
0810 23:25:08.721391 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:08.721396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:08.721466 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:08.721555 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:08.723206 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:08.724056 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:08.725718 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:08.726226 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:08.726364 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:08.728728 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:08.728733 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:08.728736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:08.728774 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:08.731377 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:08.732077 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:08.732081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:08.732084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:08.732231 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:08.732612 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:08.732616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:08.732618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:08.732976 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:08.733261 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:08.733614 (+ 353us) server_negotiation.cc:299] Negotiation successful
0810 23:25:08.733847 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 23:25:08.743713 15897 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58650:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:25:08.752997 15897 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:25:08.781585 16661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:08.770753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:46701 (local address 127.0.0.1:56192)
0810 23:25:08.772668 (+ 1915us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:08.772699 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:08.772833 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:08.773954 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:08.773964 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:08.774013 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:25:08.774577 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:08.774590 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:08.776581 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:08.776587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:08.778796 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:08.778809 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:08.780351 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:08.780357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:08.780560 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:08.780636 (+ 76us) client_negotiation.cc:769] Sending connection context
0810 23:25:08.780778 (+ 142us) client_negotiation.cc:240] Negotiation successful
0810 23:25:08.781025 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1776,"mutex_wait_us":1,"thread_start_us":1673,"threads_started":1}
I20260810 23:25:08.782423 16662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:08.771405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56192 (local address 127.14.175.65:46701)
0810 23:25:08.772237 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:08.772248 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:08.772958 (+ 710us) server_negotiation.cc:407] Connection header received
0810 23:25:08.773079 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:08.773089 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:08.773427 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:08.773631 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:25:08.774789 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:08.776418 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:08.778971 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:08.780118 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:08.780273 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:08.780521 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:08.781874 (+ 1353us) server_negotiation.cc:299] Negotiation successful
0810 23:25:08.782073 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":460,"threads_started":1}
I20260810 23:25:08.782488 16657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:08.770371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:38199 (local address 127.0.0.1:39422)
0810 23:25:08.771207 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:08.771272 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:08.771517 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:08.775249 (+ 3732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:08.775253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:08.775278 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:25:08.775694 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:08.775709 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:08.777453 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:08.777457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:08.779621 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:08.779637 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:08.781587 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:08.781593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:08.781768 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:08.781801 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 23:25:08.781866 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 23:25:08.782000 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":26,"thread_start_us":376,"threads_started":1}
I20260810 23:25:08.783182 16658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:08.770440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39422 (local address 127.14.175.66:38199)
0810 23:25:08.774770 (+ 4330us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:08.774778 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:08.774800 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:25:08.774877 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:08.774882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:08.775058 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:08.775235 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:25:08.775874 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:08.777325 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:08.779817 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:08.781325 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:08.782405 (+ 1080us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:08.782618 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:08.782690 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 23:25:08.782867 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4163,"thread_start_us":4007,"threads_started":1}
I20260810 23:25:08.783412 16663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:08.771691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:46163 (local address 127.0.0.1:56178)
0810 23:25:08.774492 (+ 2801us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:08.774513 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:08.774645 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:08.775569 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:08.775574 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:08.775604 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:25:08.776051 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:08.776064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:08.778087 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:08.778093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:08.780077 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:08.780089 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:08.782679 (+ 2590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:08.782686 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:08.782851 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:08.782884 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 23:25:08.782961 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 23:25:08.783100 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2653,"thread_start_us":2467,"threads_started":1}
I20260810 23:25:08.783766 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:08.772243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56178 (local address 127.14.175.67:46163)
0810 23:25:08.773927 (+ 1684us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:08.773937 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:08.774717 (+ 780us) server_negotiation.cc:407] Connection header received
0810 23:25:08.774829 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:08.774837 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:08.775084 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:08.775296 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:25:08.776279 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:08.777931 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:08.780240 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:08.782155 (+ 1915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:08.782287 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:08.783114 (+ 827us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:08.783203 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 23:25:08.783453 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1517,"thread_start_us":1371,"threads_started":1}
I20260810 23:25:08.787477 16326 tablet_service.cc:1467] Processing CreateTablet for tablet 5faff04870af4747b812510c9d3024c3 (DEFAULT_TABLE table=test-table [id=c7891c53c6394a979d13bd9e1c28b6bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:25:08.787410 16536 tablet_service.cc:1467] Processing CreateTablet for tablet 5faff04870af4747b812510c9d3024c3 (DEFAULT_TABLE table=test-table [id=c7891c53c6394a979d13bd9e1c28b6bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:25:08.789036 16326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5faff04870af4747b812510c9d3024c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:08.789290 16099 tablet_service.cc:1467] Processing CreateTablet for tablet 5faff04870af4747b812510c9d3024c3 (DEFAULT_TABLE table=test-table [id=c7891c53c6394a979d13bd9e1c28b6bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:25:08.790652 16099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5faff04870af4747b812510c9d3024c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:08.790980 16536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5faff04870af4747b812510c9d3024c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:08.802680 16665 tablet_bootstrap.cc:492] T 5faff04870af4747b812510c9d3024c3 P c5527e95bfdd49a98cab8c4661998c2b: Bootstrap starting.
I20260810 23:25:08.806526 16667 tablet_bootstrap.cc:492] T 5faff04870af4747b812510c9d3024c3 P 98911e534274415d832c6f424a7f7fdc: Bootstrap starting.
I20260810 23:25:08.806639 16665 tablet_bootstrap.cc:654] T 5faff04870af4747b812510c9d3024c3 P c5527e95bfdd49a98cab8c4661998c2b: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:08.807484 16669 tablet_bootstrap.cc:492] T 5faff04870af4747b812510c9d3024c3 P 908a43fd4c954206aa8900fe71b0ac36: Bootstrap starting.
I20260810 23:25:08.808139 16665 log.cc:824] T 5faff04870af4747b812510c9d3024c3 P c5527e95bfdd49a98cab8c4661998c2b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:25:08.811381 16667 tablet_bootstrap.cc:654] T 5faff04870af4747b812510c9d3024c3 P 98911e534274415d832c6f424a7f7fdc: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:08.811985 16669 tablet_bootstrap.cc:654] T 5faff04870af4747b812510c9d3024c3 P 908a43fd4c954206aa8900fe71b0ac36: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:08.812775 16667 log.cc:824] T 5faff04870af4747b812510c9d3024c3 P 98911e534274415d832c6f424a7f7fdc: Log is configured to *not* fsync() on all Append() calls
I20260810 23:25:08.813557 16669 log.cc:824] T 5faff04870af4747b812510c9d3024c3 P 908a43fd4c954206aa8900fe71b0ac36: Log is configured to *not* fsync() on all Append() calls
I20260810 23:25:08.816169 16667 tablet_bootstrap.cc:492] T 5faff04870af4747b812510c9d3024c3 P 98911e534274415d832c6f424a7f7fdc: No bootstrap required, opened a new log
I20260810 23:25:08.816500 16667 ts_tablet_manager.cc:1397] T 5faff04870af4747b812510c9d3024c3 P 98911e534274415d832c6f424a7f7fdc: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260810 23:25:08.816522 16669 tablet_bootstrap.cc:492] T 5faff04870af4747b812510c9d3024c3 P 908a43fd4c954206aa8900fe71b0ac36: No bootstrap required, opened a new log
I20260810 23:25:08.816789 16669 ts_tablet_manager.cc:1397] T 5faff04870af4747b812510c9d3024c3 P 908a43fd4c954206aa8900fe71b0ac36: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260810 23:25:08.817790 16665 tablet_bootstrap.cc:492] T 5faff04870af4747b812510c9d3024c3 P c5527e95bfdd49a98cab8c4661998c2b: No bootstrap required, opened a new log
I20260810 23:25:08.818145 16665 ts_tablet_manager.cc:1397] T 5faff04870af4747b812510c9d3024c3 P c5527e95bfdd49a98cab8c4661998c2b: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.010s
I20260810 23:25:08.825273 16669 raft_consensus.cc:348] T 5faff04870af4747b812510c9d3024c3 P 908a43fd4c954206aa8900fe71b0ac36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5527e95bfdd49a98cab8c4661998c2b" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38199 } } peers { permanent_uuid: "908a43fd4c954206aa8900fe71b0ac36" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 46163 } } peers { permanent_uuid: "98911e534274415d832c6f424a7f7fdc" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46701 } }
I20260810 23:25:08.825760 16669 raft_consensus.cc:374] T 5faff04870af4747b812510c9d3024c3 P 908a43fd4c954206aa8900fe71b0ac36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:08.825886 16669 raft_consensus.cc:729] T 5faff04870af4747b812510c9d3024c3 P 908a43fd4c954206aa8900fe71b0ac36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 908a43fd4c954206aa8900fe71b0ac36, State: Initialized, Role: FOLLOWER
I20260810 23:25:08.826598 16669 consensus_queue.cc:260] T 5faff04870af4747b812510c9d3024c3 P 908a43fd4c954206aa8900fe71b0ac36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5527e95bfdd49a98cab8c4661998c2b" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38199 } } peers { permanent_uuid: "908a43fd4c954206aa8900fe71b0ac36" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 46163 } } peers { permanent_uuid: "98911e534274415d832c6f424a7f7fdc" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46701 } }
I20260810 23:25:08.827963 16639 heartbeater.cc:503] Master 127.14.175.126:34363 was elected leader, sending a full tablet report...
I20260810 23:25:08.827723 16667 raft_consensus.cc:348] T 5faff04870af4747b812510c9d3024c3 P 98911e534274415d832c6f424a7f7fdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5527e95bfdd49a98cab8c4661998c2b" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38199 } } peers { permanent_uuid: "908a43fd4c954206aa8900fe71b0ac36" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 46163 } } peers { permanent_uuid: "98911e534274415d832c6f424a7f7fdc" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46701 } }
I20260810 23:25:08.828197 16667 raft_consensus.cc:374] T 5faff04870af4747b812510c9d3024c3 P 98911e534274415d832c6f424a7f7fdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:08.828285 16667 raft_consensus.cc:729] T 5faff04870af4747b812510c9d3024c3 P 98911e534274415d832c6f424a7f7fdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98911e534274415d832c6f424a7f7fdc, State: Initialized, Role: FOLLOWER
I20260810 23:25:08.828701 16669 ts_tablet_manager.cc:1428] T 5faff04870af4747b812510c9d3024c3 P 908a43fd4c954206aa8900fe71b0ac36: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260810 23:25:08.829068 16667 consensus_queue.cc:260] T 5faff04870af4747b812510c9d3024c3 P 98911e534274415d832c6f424a7f7fdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5527e95bfdd49a98cab8c4661998c2b" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38199 } } peers { permanent_uuid: "908a43fd4c954206aa8900fe71b0ac36" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 46163 } } peers { permanent_uuid: "98911e534274415d832c6f424a7f7fdc" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46701 } }
I20260810 23:25:08.829144 16665 raft_consensus.cc:348] T 5faff04870af4747b812510c9d3024c3 P c5527e95bfdd49a98cab8c4661998c2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5527e95bfdd49a98cab8c4661998c2b" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38199 } } peers { permanent_uuid: "908a43fd4c954206aa8900fe71b0ac36" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 46163 } } peers { permanent_uuid: "98911e534274415d832c6f424a7f7fdc" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46701 } }
I20260810 23:25:08.829646 16665 raft_consensus.cc:374] T 5faff04870af4747b812510c9d3024c3 P c5527e95bfdd49a98cab8c4661998c2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:08.829838 16665 raft_consensus.cc:729] T 5faff04870af4747b812510c9d3024c3 P c5527e95bfdd49a98cab8c4661998c2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5527e95bfdd49a98cab8c4661998c2b, State: Initialized, Role: FOLLOWER
I20260810 23:25:08.830662 16665 consensus_queue.cc:260] T 5faff04870af4747b812510c9d3024c3 P c5527e95bfdd49a98cab8c4661998c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5527e95bfdd49a98cab8c4661998c2b" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38199 } } peers { permanent_uuid: "908a43fd4c954206aa8900fe71b0ac36" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 46163 } } peers { permanent_uuid: "98911e534274415d832c6f424a7f7fdc" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46701 } }
I20260810 23:25:08.831102 16667 ts_tablet_manager.cc:1428] T 5faff04870af4747b812510c9d3024c3 P 98911e534274415d832c6f424a7f7fdc: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260810 23:25:08.832249 16430 heartbeater.cc:503] Master 127.14.175.126:34363 was elected leader, sending a full tablet report...
I20260810 23:25:08.832832 16665 ts_tablet_manager.cc:1428] T 5faff04870af4747b812510c9d3024c3 P c5527e95bfdd49a98cab8c4661998c2b: Time spent starting tablet: real 0.015s user 0.012s sys 0.002s
W20260810 23:25:08.906265 16213 tablet.cc:2369] T 5faff04870af4747b812510c9d3024c3 P 98911e534274415d832c6f424a7f7fdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:25:08.923784 16431 tablet.cc:2369] T 5faff04870af4747b812510c9d3024c3 P c5527e95bfdd49a98cab8c4661998c2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:25:08.932588 16640 tablet.cc:2369] T 5faff04870af4747b812510c9d3024c3 P 908a43fd4c954206aa8900fe71b0ac36: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:25:08.979235 16676 raft_consensus.cc:482] T 5faff04870af4747b812510c9d3024c3 P 908a43fd4c954206aa8900fe71b0ac36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:25:08.979601 16676 raft_consensus.cc:504] T 5faff04870af4747b812510c9d3024c3 P 908a43fd4c954206aa8900fe71b0ac36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5527e95bfdd49a98cab8c4661998c2b" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38199 } } peers { permanent_uuid: "908a43fd4c954206aa8900fe71b0ac36" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 46163 } } peers { permanent_uuid: "98911e534274415d832c6f424a7f7fdc" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46701 } }
I20260810 23:25:08.982124 16676 leader_election.cc:288] T 5faff04870af4747b812510c9d3024c3 P 908a43fd4c954206aa8900fe71b0ac36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5527e95bfdd49a98cab8c4661998c2b (127.14.175.66:38199), 98911e534274415d832c6f424a7f7fdc (127.14.175.65:46701)
I20260810 23:25:08.988665 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:08.982331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:38199 (local address 127.14.175.67:52645)
0810 23:25:08.982607 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:08.982627 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:08.982767 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:08.983559 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:08.983565 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:08.983592 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:25:08.983812 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:08.983830 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:08.985084 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:08.985090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:08.986653 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:08.986666 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:08.987923 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:08.987934 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:08.988134 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:08.988170 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 23:25:08.988262 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 23:25:08.988415 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":1}
I20260810 23:25:08.988705 16658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:08.982503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:52645 (local address 127.14.175.66:38199)
0810 23:25:08.982734 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:08.982740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:08.982764 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:25:08.982970 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:08.982976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:08.983060 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:08.983142 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:25:08.984043 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:08.984921 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:08.986832 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:08.987749 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:08.987873 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:08.987929 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:08.988292 (+ 363us) server_negotiation.cc:299] Negotiation successful
0810 23:25:08.988421 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 23:25:08.989068 16662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:08.982854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:50845 (local address 127.14.175.65:46701)
0810 23:25:08.983134 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:08.983140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:08.983437 (+ 297us) server_negotiation.cc:407] Connection header received
0810 23:25:08.983541 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:08.983546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:08.983621 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:08.983724 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:25:08.984386 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:08.985319 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:08.987231 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:08.988114 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:08.988280 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:08.988348 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:08.988682 (+ 334us) server_negotiation.cc:299] Negotiation successful
0810 23:25:08.988850 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 23:25:08.989113 16683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:08.982544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:46701 (local address 127.14.175.67:50845)
0810 23:25:08.983297 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:08.983312 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:08.983426 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:08.983747 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:08.983750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:08.983770 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:25:08.984210 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:08.984223 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:08.985481 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:08.985485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:08.987056 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:08.987070 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:08.988329 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:08.988336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:08.988503 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:08.988542 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 23:25:08.988636 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 23:25:08.988768 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":1,"thread_start_us":454,"threads_started":1}
I20260810 23:25:08.990316 16354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5faff04870af4747b812510c9d3024c3" candidate_uuid: "908a43fd4c954206aa8900fe71b0ac36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5527e95bfdd49a98cab8c4661998c2b" is_pre_election: true
I20260810 23:25:08.990327 16137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5faff04870af4747b812510c9d3024c3" candidate_uuid: "908a43fd4c954206aa8900fe71b0ac36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98911e534274415d832c6f424a7f7fdc" is_pre_election: true
I20260810 23:25:08.991043 16354 raft_consensus.cc:2454] T 5faff04870af4747b812510c9d3024c3 P c5527e95bfdd49a98cab8c4661998c2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 908a43fd4c954206aa8900fe71b0ac36 in term 0.
I20260810 23:25:08.991852 16461 leader_election.cc:302] T 5faff04870af4747b812510c9d3024c3 P 908a43fd4c954206aa8900fe71b0ac36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 908a43fd4c954206aa8900fe71b0ac36, c5527e95bfdd49a98cab8c4661998c2b; no voters:
I20260810 23:25:08.992259 16676 raft_consensus.cc:2781] T 5faff04870af4747b812510c9d3024c3 P 908a43fd4c954206aa8900fe71b0ac36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:25:08.992393 16676 raft_consensus.cc:482] T 5faff04870af4747b812510c9d3024c3 P 908a43fd4c954206aa8900fe71b0ac36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:25:08.992499 16676 raft_consensus.cc:3037] T 5faff04870af4747b812510c9d3024c3 P 908a43fd4c954206aa8900fe71b0ac36 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:08.993309 16137 raft_consensus.cc:2454] T 5faff04870af4747b812510c9d3024c3 P 98911e534274415d832c6f424a7f7fdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 908a43fd4c954206aa8900fe71b0ac36 in term 0.
I20260810 23:25:08.995333 16676 raft_consensus.cc:504] T 5faff04870af4747b812510c9d3024c3 P 908a43fd4c954206aa8900fe71b0ac36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5527e95bfdd49a98cab8c4661998c2b" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38199 } } peers { permanent_uuid: "908a43fd4c954206aa8900fe71b0ac36" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 46163 } } peers { permanent_uuid: "98911e534274415d832c6f424a7f7fdc" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46701 } }
I20260810 23:25:08.996023 16676 leader_election.cc:288] T 5faff04870af4747b812510c9d3024c3 P 908a43fd4c954206aa8900fe71b0ac36 [CANDIDATE]: Term 1 election: Requested vote from peers c5527e95bfdd49a98cab8c4661998c2b (127.14.175.66:38199), 98911e534274415d832c6f424a7f7fdc (127.14.175.65:46701)
I20260810 23:25:08.996549 16354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5faff04870af4747b812510c9d3024c3" candidate_uuid: "908a43fd4c954206aa8900fe71b0ac36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5527e95bfdd49a98cab8c4661998c2b"
I20260810 23:25:08.996574 16137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5faff04870af4747b812510c9d3024c3" candidate_uuid: "908a43fd4c954206aa8900fe71b0ac36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98911e534274415d832c6f424a7f7fdc"
I20260810 23:25:08.996757 16354 raft_consensus.cc:3037] T 5faff04870af4747b812510c9d3024c3 P c5527e95bfdd49a98cab8c4661998c2b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:08.996814 16137 raft_consensus.cc:3037] T 5faff04870af4747b812510c9d3024c3 P 98911e534274415d832c6f424a7f7fdc [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:08.999150 16354 raft_consensus.cc:2454] T 5faff04870af4747b812510c9d3024c3 P c5527e95bfdd49a98cab8c4661998c2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 908a43fd4c954206aa8900fe71b0ac36 in term 1.
I20260810 23:25:08.999603 16461 leader_election.cc:302] T 5faff04870af4747b812510c9d3024c3 P 908a43fd4c954206aa8900fe71b0ac36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 908a43fd4c954206aa8900fe71b0ac36, c5527e95bfdd49a98cab8c4661998c2b; no voters:
I20260810 23:25:08.999907 16676 raft_consensus.cc:2781] T 5faff04870af4747b812510c9d3024c3 P 908a43fd4c954206aa8900fe71b0ac36 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:25:09.000094 16137 raft_consensus.cc:2454] T 5faff04870af4747b812510c9d3024c3 P 98911e534274415d832c6f424a7f7fdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 908a43fd4c954206aa8900fe71b0ac36 in term 1.
I20260810 23:25:09.000506 16676 raft_consensus.cc:686] T 5faff04870af4747b812510c9d3024c3 P 908a43fd4c954206aa8900fe71b0ac36 [term 1 LEADER]: Becoming Leader. State: Replica: 908a43fd4c954206aa8900fe71b0ac36, State: Running, Role: LEADER
I20260810 23:25:09.001065 16676 consensus_queue.cc:237] T 5faff04870af4747b812510c9d3024c3 P 908a43fd4c954206aa8900fe71b0ac36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5527e95bfdd49a98cab8c4661998c2b" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38199 } } peers { permanent_uuid: "908a43fd4c954206aa8900fe71b0ac36" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 46163 } } peers { permanent_uuid: "98911e534274415d832c6f424a7f7fdc" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46701 } }
I20260810 23:25:09.007206 15901 catalog_manager.cc:5495] T 5faff04870af4747b812510c9d3024c3 P 908a43fd4c954206aa8900fe71b0ac36 reported cstate change: term changed from 0 to 1, leader changed from <none> to 908a43fd4c954206aa8900fe71b0ac36 (127.14.175.67). New cstate: current_term: 1 leader_uuid: "908a43fd4c954206aa8900fe71b0ac36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5527e95bfdd49a98cab8c4661998c2b" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "908a43fd4c954206aa8900fe71b0ac36" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 46163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98911e534274415d832c6f424a7f7fdc" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46701 } health_report { overall_health: UNKNOWN } } }
Aug 10 23:25:09 dist-test-slave-rmhg krb5kdc[15828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.65@KRBTEST.COM
I20260810 23:25:09.082482 16662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:09.068193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56198 (local address 127.14.175.65:46701)
0810 23:25:09.068433 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:09.068441 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:09.068888 (+ 447us) server_negotiation.cc:407] Connection header received
0810 23:25:09.069023 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:09.069027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:09.069095 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:09.069228 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:09.070734 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:09.071541 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:09.072797 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:09.073237 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:09.073327 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:09.077276 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:09.077294 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:09.077313 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:09.077351 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:09.079980 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:09.080626 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:09.080633 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:09.080635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:09.080772 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:09.081059 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:09.081062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:09.081064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:09.081475 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:09.081700 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:09.081948 (+ 248us) server_negotiation.cc:299] Negotiation successful
0810 23:25:09.082125 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
Aug 10 23:25:09 dist-test-slave-rmhg krb5kdc[15828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404309, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 23:25:09 dist-test-slave-rmhg krb5kdc[15828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404309, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.175.65@KRBTEST.COM
I20260810 23:25:09.133419 16662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:09.118989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56206 (local address 127.14.175.65:46701)
0810 23:25:09.119211 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:09.119218 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:09.119667 (+ 449us) server_negotiation.cc:407] Connection header received
0810 23:25:09.119808 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:09.119812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:09.119885 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:09.119977 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:09.121557 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:09.122348 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:09.123729 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:09.124245 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:09.124355 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:09.128451 (+ 4096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:09.128457 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:09.128461 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:09.128504 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:09.131135 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:09.131808 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:09.131812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:09.131814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:09.131949 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:09.132239 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:09.132242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:09.132244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:09.132515 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:09.132728 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:09.132890 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 23:25:09.133071 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Aug 10 23:25:09 dist-test-slave-rmhg krb5kdc[15828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 23:25:09 dist-test-slave-rmhg krb5kdc[15828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.65@KRBTEST.COM
I20260810 23:25:09.187186 16662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:09.169631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56214 (local address 127.14.175.65:46701)
0810 23:25:09.169870 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:09.169876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:09.170291 (+ 415us) server_negotiation.cc:407] Connection header received
0810 23:25:09.170527 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:09.170536 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:09.170614 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:09.170711 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:09.172700 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:09.173495 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:09.175379 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:09.175857 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:09.175974 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:09.180990 (+ 5016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:09.180996 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:09.181000 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:09.181043 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:09.184274 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:09.185120 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:09.185126 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:09.185128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:09.185329 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:09.185718 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:09.185722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:09.185724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:09.186078 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:09.186270 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:09.186588 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 23:25:09.186823 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
W20260810 23:25:09.189646 16062 tablet_service.cc:2908] Rejecting scan request for tablet 5faff04870af4747b812510c9d3024c3: Uninitialized: clean time has not yet been initialized
I20260810 23:25:09.193499 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 15997
I20260810 23:25:09.206961 16662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:09.194534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56222 (local address 127.14.175.65:46701)
0810 23:25:09.194706 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:09.194710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:09.195001 (+ 291us) server_negotiation.cc:407] Connection header received
0810 23:25:09.195180 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:09.195184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:09.195263 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:09.195344 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:09.196764 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:09.197552 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:09.198801 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:09.199248 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:09.199368 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:09.201692 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:09.201696 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:09.201699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:09.201731 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:09.204185 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:09.205003 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:09.205008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:09.205010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:09.205218 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:09.205660 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:09.205663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:09.205665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:09.205957 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:09.206212 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:09.206494 (+ 282us) server_negotiation.cc:299] Negotiation successful
0810 23:25:09.206694 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:25:09.207700 16204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:25:09.302436 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 15997
I20260810 23:25:09.338243 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 16220
Aug 10 23:25:09 dist-test-slave-rmhg krb5kdc[15828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.66@KRBTEST.COM
I20260810 23:25:09.353605 16658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:09.339115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39426 (local address 127.14.175.66:38199)
0810 23:25:09.339303 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:09.339309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:09.339347 (+ 38us) server_negotiation.cc:407] Connection header received
0810 23:25:09.339411 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:09.339415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:09.339491 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:09.339600 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:09.340823 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:09.341651 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:09.343070 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:09.343506 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:09.343638 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:09.347955 (+ 4317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:09.347974 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:09.347993 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:09.348038 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:09.350882 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:09.351593 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:09.351601 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:09.351603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:09.351766 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:09.352111 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:09.352114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:09.352116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:09.352493 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:09.352787 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:09.353020 (+ 233us) server_negotiation.cc:299] Negotiation successful
0810 23:25:09.353261 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:25:09.354421 16423 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:25:09.441924 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 16220
W20260810 23:25:09.464758 16461 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.175.66:38199 (error 108)
I20260810 23:25:09.465241 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 16436
I20260810 23:25:09.466832 16683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:09.466227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:38199 (local address 127.14.175.67:35161)
0810 23:25:09.466386 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:09.466615 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.66:38199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 23:25:09.467157 16461 consensus_peers.cc:487] T 5faff04870af4747b812510c9d3024c3 P 908a43fd4c954206aa8900fe71b0ac36 -> Peer c5527e95bfdd49a98cab8c4661998c2b (127.14.175.66:38199): Couldn't send request to peer c5527e95bfdd49a98cab8c4661998c2b. Status: Network error: Client connection negotiation failed: client connection to 127.14.175.66:38199: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
Aug 10 23:25:09 dist-test-slave-rmhg krb5kdc[15828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.67@KRBTEST.COM
I20260810 23:25:09.481462 16707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:09.466098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56188 (local address 127.14.175.67:46163)
0810 23:25:09.466763 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:09.466769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:09.466835 (+ 66us) server_negotiation.cc:407] Connection header received
0810 23:25:09.466905 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:09.466910 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:09.466989 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:09.467113 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:09.468504 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:09.469315 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:09.470669 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:09.471113 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:09.471232 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:09.475587 (+ 4355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:09.475603 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:09.475621 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:09.475663 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:09.478489 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:09.479216 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:09.479224 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:09.479226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:09.479380 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:09.479794 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:09.479798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:09.479800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:09.480180 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:09.480477 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:09.480819 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 23:25:09.481050 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":432,"threads_started":1}
I20260810 23:25:09.482146 16632 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:25:09.571364 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 16436
I20260810 23:25:09.595077 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 15849
I20260810 23:25:09.596292 15948 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:25:09.707880 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 15849
2026-08-10T23:25:09Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4608 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:25:09 dist-test-slave-rmhg krb5kdc[16715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:25:09 dist-test-slave-rmhg krb5kdc[16715](info): set up 2 sockets
Aug 10 23:25:09 dist-test-slave-rmhg krb5kdc[16715](info): commencing operation
krb5kdc: starting...
W20260810 23:25:10.928646 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.171s user 0.005s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:25:11 dist-test-slave-rmhg krb5kdc[16715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:25:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:25:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
I20260810 23:25:11.141359 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:41753
--webserver_interface=127.14.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:33397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:41753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:11.499140 16740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:11.499511 16740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:11.499668 16740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:11.507297 16740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:25:11.507458 16740 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 23:25:11.507524 16740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:11.507620 16740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:11.507695 16740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:25:11.507782 16740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:25:11.518342 16740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:33397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:41753
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:41753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:11.519927 16740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:11.521488 16740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:11.534574 16747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:11.534498 16749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:11.534628 16752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:11.535341 16740 server_base.cc:1034] running on GCE node
I20260810 23:25:11.536258 16740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:11.538009 16740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:11.539338 16740 hybrid_clock.cc:648] HybridClock initialized: now 1786404311539253 us; error 49 us; skew 500 ppm
I20260810 23:25:11.539820 16740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:11 dist-test-slave-rmhg krb5kdc[16715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404311, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:11.543653 16740 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:25:11.545584 16740 webserver.cc:460] Webserver started at http://127.14.175.126:43481/ using document root <none> and password file <none>
I20260810 23:25:11.546404 16740 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:11.546533 16740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:11.546907 16740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:11.550629 16740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "cf20257f3724473b8ce824cb8fa6f987"
format_stamp: "Formatted at 2026-08-10 23:25:11 on dist-test-slave-rmhg"
server_key: "3bde8528a8afa9954347249c18bccd6f"
server_key_iv: "c4d98899652aa99c2a7848a445c69fa1"
server_key_version: "encryptionkey@0"
I20260810 23:25:11.551550 16740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "cf20257f3724473b8ce824cb8fa6f987"
format_stamp: "Formatted at 2026-08-10 23:25:11 on dist-test-slave-rmhg"
server_key: "3bde8528a8afa9954347249c18bccd6f"
server_key_iv: "c4d98899652aa99c2a7848a445c69fa1"
server_key_version: "encryptionkey@0"
I20260810 23:25:11.557953 16740 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260810 23:25:11.562774 16764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:11.564570 16740 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260810 23:25:11.564807 16740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "cf20257f3724473b8ce824cb8fa6f987"
format_stamp: "Formatted at 2026-08-10 23:25:11 on dist-test-slave-rmhg"
server_key: "3bde8528a8afa9954347249c18bccd6f"
server_key_iv: "c4d98899652aa99c2a7848a445c69fa1"
server_key_version: "encryptionkey@0"
I20260810 23:25:11.565068 16740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:11.596972 16740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:11.602048 16740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:11.602347 16740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:11.625842 16740 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:41753
I20260810 23:25:11.625890 16855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:41753 every 8 connection(s)
I20260810 23:25:11.627416 16740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:25:11.631692 16859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:11.633899 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 16740
I20260810 23:25:11.634135 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:25:11.634806 15037 external_mini_cluster.cc:1428] Setting key 11f4af02828583bf696d0eb63296e745
I20260810 23:25:11.645794 16859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf20257f3724473b8ce824cb8fa6f987: Bootstrap starting.
I20260810 23:25:11.649739 16859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf20257f3724473b8ce824cb8fa6f987: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:11.650902 16859 log.cc:824] T 00000000000000000000000000000000 P cf20257f3724473b8ce824cb8fa6f987: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:25:11 dist-test-slave-rmhg krb5kdc[16715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:11.654232 16859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf20257f3724473b8ce824cb8fa6f987: No bootstrap required, opened a new log
I20260810 23:25:11.662025 16864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:11.637260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47362 (local address 127.14.175.126:41753)
0810 23:25:11.638184 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:11.638200 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:11.638253 (+ 53us) server_negotiation.cc:407] Connection header received
0810 23:25:11.640732 (+ 2479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:11.640766 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:11.641400 (+ 634us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:11.641938 (+ 538us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:11.643341 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:11.645357 (+ 2016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:11.646812 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:11.647963 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:11.648201 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:11.652716 (+ 4515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:11.652725 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:11.652744 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:11.652784 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:11.656513 (+ 3729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:11.657377 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:11.657387 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:11.657397 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:11.657657 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:11.658097 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:11.658103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:11.658106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:11.658993 (+ 887us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:11.659303 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:11.659820 (+ 517us) server_negotiation.cc:299] Negotiation successful
0810 23:25:11.660358 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":409,"threads_started":1}
I20260810 23:25:11.664569 16859 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf20257f3724473b8ce824cb8fa6f987 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf20257f3724473b8ce824cb8fa6f987" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 41753 } }
I20260810 23:25:11.664964 16859 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf20257f3724473b8ce824cb8fa6f987 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:11.665134 16859 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf20257f3724473b8ce824cb8fa6f987 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf20257f3724473b8ce824cb8fa6f987, State: Initialized, Role: FOLLOWER
I20260810 23:25:11.665866 16859 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf20257f3724473b8ce824cb8fa6f987 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf20257f3724473b8ce824cb8fa6f987" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 41753 } }
I20260810 23:25:11.666114 16859 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf20257f3724473b8ce824cb8fa6f987 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:25:11.666222 16859 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf20257f3724473b8ce824cb8fa6f987 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:25:11.666353 16859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf20257f3724473b8ce824cb8fa6f987 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:11.668625 16859 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf20257f3724473b8ce824cb8fa6f987 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf20257f3724473b8ce824cb8fa6f987" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 41753 } }
I20260810 23:25:11.669293 16859 leader_election.cc:302] T 00000000000000000000000000000000 P cf20257f3724473b8ce824cb8fa6f987 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf20257f3724473b8ce824cb8fa6f987; no voters:
I20260810 23:25:11.669971 16859 leader_election.cc:288] T 00000000000000000000000000000000 P cf20257f3724473b8ce824cb8fa6f987 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:25:11.670091 16867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf20257f3724473b8ce824cb8fa6f987 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:25:11.670809 16867 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf20257f3724473b8ce824cb8fa6f987 [term 1 LEADER]: Becoming Leader. State: Replica: cf20257f3724473b8ce824cb8fa6f987, State: Running, Role: LEADER
I20260810 23:25:11.671440 16867 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf20257f3724473b8ce824cb8fa6f987 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf20257f3724473b8ce824cb8fa6f987" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 41753 } }
I20260810 23:25:11.672547 16859 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf20257f3724473b8ce824cb8fa6f987 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:25:11.674615 16871 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf20257f3724473b8ce824cb8fa6f987 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf20257f3724473b8ce824cb8fa6f987" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf20257f3724473b8ce824cb8fa6f987" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 41753 } } }
I20260810 23:25:11.674788 16869 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf20257f3724473b8ce824cb8fa6f987 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf20257f3724473b8ce824cb8fa6f987. Latest consensus state: current_term: 1 leader_uuid: "cf20257f3724473b8ce824cb8fa6f987" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf20257f3724473b8ce824cb8fa6f987" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 41753 } } }
I20260810 23:25:11.675210 16871 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf20257f3724473b8ce824cb8fa6f987 [sys.catalog]: This master's current role is: LEADER
I20260810 23:25:11.675472 16869 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf20257f3724473b8ce824cb8fa6f987 [sys.catalog]: This master's current role is: LEADER
I20260810 23:25:11.677218 16881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:25:11.683676 16881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:25:11.694288 16881 catalog_manager.cc:1347] Generated new cluster ID: fc2b2f650c7d4d159c0f5356a497ef75
I20260810 23:25:11.694427 16881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:25:11.704598 16881 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:25:11.706943 16881 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:25:11.722604 16881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf20257f3724473b8ce824cb8fa6f987: Generated new TSK 0
I20260810 23:25:11.723565 16881 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
I20260810 23:25:11.785957 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.65:0
--local_ip_for_outbound_sockets=127.14.175.65
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:41753
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:33397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:12.145771 16901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:12.146088 16901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:12.146219 16901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:12.154031 16901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:12.154161 16901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:12.154273 16901 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:25:12.154383 16901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.65
I20260810 23:25:12.164986 16901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:33397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.175.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.175.126:41753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.65
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:12.166433 16901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:12.167853 16901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:12.181916 16911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:12.182123 16913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:12.182276 16918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:12.183398 16901 server_base.cc:1034] running on GCE node
I20260810 23:25:12.184057 16901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:12.185114 16901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:12.186398 16901 hybrid_clock.cc:648] HybridClock initialized: now 1786404312186321 us; error 47 us; skew 500 ppm
I20260810 23:25:12.186735 16901 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:12 dist-test-slave-rmhg krb5kdc[16715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404312, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:12.190135 16901 init.cc:377] Logged in from keytab as kudu/127.14.175.65@KRBTEST.COM (short username kudu)
I20260810 23:25:12.191715 16901 webserver.cc:460] Webserver started at http://127.14.175.65:37455/ using document root <none> and password file <none>
I20260810 23:25:12.192448 16901 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:12.192556 16901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:12.192888 16901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:12.195961 16901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-0/data/instance:
uuid: "aacd1a6c649b47e78f3a71c94a3f4437"
format_stamp: "Formatted at 2026-08-10 23:25:12 on dist-test-slave-rmhg"
server_key: "7748ba5890e25cae5cdb4a3234b34aad"
server_key_iv: "9cd2e988a8e26f5469f681c9f5932339"
server_key_version: "encryptionkey@0"
I20260810 23:25:12.196791 16901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance:
uuid: "aacd1a6c649b47e78f3a71c94a3f4437"
format_stamp: "Formatted at 2026-08-10 23:25:12 on dist-test-slave-rmhg"
server_key: "7748ba5890e25cae5cdb4a3234b34aad"
server_key_iv: "9cd2e988a8e26f5469f681c9f5932339"
server_key_version: "encryptionkey@0"
I20260810 23:25:12.202221 16901 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 23:25:12.206274 16931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:12.207851 16901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:25:12.208043 16901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-0/wal
uuid: "aacd1a6c649b47e78f3a71c94a3f4437"
format_stamp: "Formatted at 2026-08-10 23:25:12 on dist-test-slave-rmhg"
server_key: "7748ba5890e25cae5cdb4a3234b34aad"
server_key_iv: "9cd2e988a8e26f5469f681c9f5932339"
server_key_version: "encryptionkey@0"
I20260810 23:25:12.208280 16901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:12.232000 16901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:12.236873 16901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:12.237205 16901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:12.240279 16901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:25:12.240382 16901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:12.240525 16901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:25:12.240613 16901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:25:12 dist-test-slave-rmhg krb5kdc[16715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404312, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:12.266134 16944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:12.241414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:54095 (local address 127.14.175.126:41753)
0810 23:25:12.241877 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:12.241882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:12.243418 (+ 1536us) server_negotiation.cc:407] Connection header received
0810 23:25:12.245733 (+ 2315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:12.245741 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:12.245843 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:12.245929 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:12.248932 (+ 3003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:12.249966 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:12.252600 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:12.253257 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:12.253412 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:12.258539 (+ 5127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:12.258546 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:12.258549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:12.258598 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:12.262096 (+ 3498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:12.263335 (+ 1239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:12.263342 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:12.263346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:12.263523 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:12.264154 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:12.264160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:12.264164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:12.264471 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:12.264608 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:12.265499 (+ 891us) server_negotiation.cc:299] Negotiation successful
0810 23:25:12.265750 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":246,"threads_started":1}
I20260810 23:25:12.267568 16945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:12.242220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:41753 (local address 127.14.175.65:54095)
0810 23:25:12.243184 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:12.243249 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:12.245320 (+ 2071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:12.246561 (+ 1241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:12.246583 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:12.247527 (+ 944us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:12.248592 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:12.248634 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:12.250260 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:12.250269 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:12.252338 (+ 2069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:12.252366 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:12.253620 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:12.253633 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:12.254009 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:12.254913 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:12.254946 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:12.258233 (+ 3287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:12.262328 (+ 4095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:12.262341 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:12.262368 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:12.263177 (+ 809us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:12.263706 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:12.263712 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:12.263715 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:12.264022 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:12.264626 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:12.264637 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:12.265053 (+ 416us) client_negotiation.cc:769] Sending connection context
0810 23:25:12.265389 (+ 336us) client_negotiation.cc:240] Negotiation successful
0810 23:25:12.266040 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":392,"threads_started":1}
I20260810 23:25:12.290258 16901 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.65:46539
I20260810 23:25:12.290280 17118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.65:46539 every 8 connection(s)
I20260810 23:25:12.291913 16901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
I20260810 23:25:12.294639 17119 heartbeater.cc:347] Connected to a master server at 127.14.175.126:41753
I20260810 23:25:12.294907 17119 heartbeater.cc:464] Registering TS with master...
I20260810 23:25:12.295816 17119 heartbeater.cc:511] Master 127.14.175.126:41753 requested a full tablet report, sending...
I20260810 23:25:12.298465 16791 ts_manager.cc:194] Registered new tserver with Master: aacd1a6c649b47e78f3a71c94a3f4437 (127.14.175.65:46539)
I20260810 23:25:12.300707 16791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.65@KRBTEST.COM'} at 127.14.175.65:54095
I20260810 23:25:12.301041 17104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:12.284949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:41753 (local address 127.14.175.65:35027)
0810 23:25:12.285749 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:12.285768 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:12.285939 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:12.286387 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:12.286394 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:12.287209 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:12.287809 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:12.287833 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:12.289114 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:12.289122 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:12.291042 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:12.291061 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:12.291920 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:12.291928 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:12.292118 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:12.292802 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:12.292831 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:12.294280 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:12.298099 (+ 3819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:12.298104 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:12.298107 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:12.298544 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:12.299048 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:12.299052 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:12.299055 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:12.299197 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:12.299969 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:12.299972 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:12.300162 (+ 190us) client_negotiation.cc:769] Sending connection context
0810 23:25:12.300275 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 23:25:12.300575 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":520,"threads_started":1}
I20260810 23:25:12.301051 16944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:12.284745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:35027 (local address 127.14.175.126:41753)
0810 23:25:12.284961 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:12.284968 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:12.285884 (+ 916us) server_negotiation.cc:407] Connection header received
0810 23:25:12.286081 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:12.286088 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:12.286194 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:12.286307 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:12.288042 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:12.288920 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:12.291227 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:12.291699 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:12.291815 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:12.294469 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:12.294474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:12.294478 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:12.294513 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:12.297914 (+ 3401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:12.298705 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:12.298712 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:12.298715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:12.298889 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:12.299356 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:12.299361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:12.299364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:12.299840 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:12.300009 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:12.300423 (+ 414us) server_negotiation.cc:299] Negotiation successful
0810 23:25:12.300678 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":79}
I20260810 23:25:12.301118 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 16901
I20260810 23:25:12.301368 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance
I20260810 23:25:12.302168 15037 external_mini_cluster.cc:1428] Setting key 5d629072bac8768476f160181e996087
WARNING: no policy specified for kudu/127.14.175.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
I20260810 23:25:12.346210 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.66:0
--local_ip_for_outbound_sockets=127.14.175.66
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:41753
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:33397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:12.697306 17123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:12.697655 17123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:12.697809 17123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:12.707391 17123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:12.707576 17123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:12.707716 17123 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:25:12.707823 17123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.66
I20260810 23:25:12.721951 17123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:33397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.175.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.175.126:41753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.66
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:12.723356 17123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:12.724808 17123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:12.739112 17131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:12.739926 17134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:12.739948 17132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:12.740440 17123 server_base.cc:1034] running on GCE node
I20260810 23:25:12.741178 17123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:12.742244 17123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:12.743501 17123 hybrid_clock.cc:648] HybridClock initialized: now 1786404312743405 us; error 58 us; skew 500 ppm
I20260810 23:25:12.743821 17123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:12 dist-test-slave-rmhg krb5kdc[16715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404312, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:12.747319 17123 init.cc:377] Logged in from keytab as kudu/127.14.175.66@KRBTEST.COM (short username kudu)
I20260810 23:25:12.749020 17123 webserver.cc:460] Webserver started at http://127.14.175.66:37105/ using document root <none> and password file <none>
I20260810 23:25:12.749831 17123 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:12.749946 17123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:12.750296 17123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:12.753449 17123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-1/data/instance:
uuid: "d56e501300134723906fdfeb8637f97e"
format_stamp: "Formatted at 2026-08-10 23:25:12 on dist-test-slave-rmhg"
server_key: "f84ebc8f36ec6d23ef05105b099d938c"
server_key_iv: "d7217fad33855fd672ce1f8ab65235db"
server_key_version: "encryptionkey@0"
I20260810 23:25:12.754305 17123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance:
uuid: "d56e501300134723906fdfeb8637f97e"
format_stamp: "Formatted at 2026-08-10 23:25:12 on dist-test-slave-rmhg"
server_key: "f84ebc8f36ec6d23ef05105b099d938c"
server_key_iv: "d7217fad33855fd672ce1f8ab65235db"
server_key_version: "encryptionkey@0"
I20260810 23:25:12.759891 17123 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:25:12.763814 17141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:12.765370 17123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:25:12.765547 17123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-1/wal
uuid: "d56e501300134723906fdfeb8637f97e"
format_stamp: "Formatted at 2026-08-10 23:25:12 on dist-test-slave-rmhg"
server_key: "f84ebc8f36ec6d23ef05105b099d938c"
server_key_iv: "d7217fad33855fd672ce1f8ab65235db"
server_key_version: "encryptionkey@0"
I20260810 23:25:12.765772 17123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:12.792083 17123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:12.797114 17123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:12.797462 17123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:12.800529 17123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:25:12.800648 17123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:12.800797 17123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:25:12.800945 17123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 10 23:25:12 dist-test-slave-rmhg krb5kdc[16715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404312, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:12.826820 16944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:12.800990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:43665 (local address 127.14.175.126:41753)
0810 23:25:12.801236 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:12.801243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:12.803005 (+ 1762us) server_negotiation.cc:407] Connection header received
0810 23:25:12.805271 (+ 2266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:12.805277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:12.805374 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:12.805471 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:12.808097 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:12.809207 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:12.812659 (+ 3452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:12.813415 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:12.813574 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:12.819015 (+ 5441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:12.819021 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:12.819025 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:12.819067 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:12.822769 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:12.823929 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:12.823935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:12.823937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:12.824090 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:12.824609 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:12.824614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:12.824617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:12.824931 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:12.825103 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:12.826198 (+ 1095us) server_negotiation.cc:299] Negotiation successful
0810 23:25:12.826458 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 23:25:12.828012 17154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:12.801903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:41753 (local address 127.14.175.66:43665)
0810 23:25:12.802802 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:12.802857 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:12.804843 (+ 1986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:12.805825 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:12.805840 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:12.806769 (+ 929us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:12.807835 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:12.807877 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:12.809456 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:12.809463 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:12.812327 (+ 2864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:12.812352 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:12.813745 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:12.813759 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:12.814052 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:12.814959 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:12.815004 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:12.818674 (+ 3670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:12.822993 (+ 4319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:12.823006 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:12.823032 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:12.823774 (+ 742us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:12.824227 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:12.824231 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:12.824233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:12.824482 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:12.825086 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:12.825095 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:12.825677 (+ 582us) client_negotiation.cc:769] Sending connection context
0810 23:25:12.826069 (+ 392us) client_negotiation.cc:240] Negotiation successful
0810 23:25:12.826708 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":344,"threads_started":1}
I20260810 23:25:12.850359 16944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:12.834898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:60289 (local address 127.14.175.126:41753)
0810 23:25:12.835139 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:12.835146 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:12.835926 (+ 780us) server_negotiation.cc:407] Connection header received
0810 23:25:12.836201 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:12.836207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:12.836281 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:12.836367 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:12.837519 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:12.838580 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:12.840494 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:12.841111 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:12.841293 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:12.844017 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:12.844025 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:12.844031 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:12.844086 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:12.847296 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:12.848188 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:12.848192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:12.848194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:12.848342 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:12.848715 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:12.848718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:12.848721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:12.849198 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:12.849360 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:12.849925 (+ 565us) server_negotiation.cc:299] Negotiation successful
0810 23:25:12.850121 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 23:25:12.850786 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:12.834478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:41753 (local address 127.14.175.66:60289)
0810 23:25:12.835768 (+ 1290us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:12.835794 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:12.836050 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:12.836424 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:12.836428 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:12.836952 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:12.837348 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:12.837362 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:12.838730 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:12.838736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:12.840318 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:12.840345 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:12.841395 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:12.841403 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:12.841614 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:12.842306 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:12.842331 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:12.843688 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:12.847448 (+ 3760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:12.847454 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:12.847459 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:12.848068 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:12.848436 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:12.848441 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:12.848445 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:12.848613 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:12.849345 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:12.849350 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:12.849615 (+ 265us) client_negotiation.cc:769] Sending connection context
0810 23:25:12.849833 (+ 218us) client_negotiation.cc:240] Negotiation successful
0810 23:25:12.850245 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1117,"thread_start_us":666,"threads_started":1}
I20260810 23:25:12.854292 17123 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.66:38183
I20260810 23:25:12.854305 17363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.66:38183 every 8 connection(s)
I20260810 23:25:12.855579 17123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
I20260810 23:25:12.857914 17365 heartbeater.cc:347] Connected to a master server at 127.14.175.126:41753
I20260810 23:25:12.858157 17365 heartbeater.cc:464] Registering TS with master...
I20260810 23:25:12.858876 17365 heartbeater.cc:511] Master 127.14.175.126:41753 requested a full tablet report, sending...
I20260810 23:25:12.859491 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 17123
I20260810 23:25:12.859673 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance
I20260810 23:25:12.860162 15037 external_mini_cluster.cc:1428] Setting key d26496a51cc64709c52f3a7123b7b9a6
I20260810 23:25:12.860318 16791 ts_manager.cc:194] Registered new tserver with Master: d56e501300134723906fdfeb8637f97e (127.14.175.66:38183)
I20260810 23:25:12.861215 16791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.66@KRBTEST.COM'} at 127.14.175.66:43665
WARNING: no policy specified for kudu/127.14.175.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
I20260810 23:25:12.905001 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.67:0
--local_ip_for_outbound_sockets=127.14.175.67
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:41753
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:33397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:13.255515 17371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:13.255832 17371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:13.255959 17371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:13.263518 17371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:13.263654 17371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:13.263784 17371 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:25:13.263892 17371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.67
I20260810 23:25:13.274559 17371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:33397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.175.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.175.126:41753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.67
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:13.275930 17371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:13.277350 17371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:13.290962 17377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:13.291145 17378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:13.290956 17380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:13.292872 17371 server_base.cc:1034] running on GCE node
I20260810 23:25:13.293641 17371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:13.294704 17371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:13.295966 17371 hybrid_clock.cc:648] HybridClock initialized: now 1786404313295901 us; error 40 us; skew 500 ppm
I20260810 23:25:13.296294 17371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:13 dist-test-slave-rmhg krb5kdc[16715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404313, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:13.299604 17371 init.cc:377] Logged in from keytab as kudu/127.14.175.67@KRBTEST.COM (short username kudu)
I20260810 23:25:13.301257 17371 webserver.cc:460] Webserver started at http://127.14.175.67:46057/ using document root <none> and password file <none>
I20260810 23:25:13.302011 17371 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:13.302117 17371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:13.302456 17371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:13.305462 17119 heartbeater.cc:503] Master 127.14.175.126:41753 was elected leader, sending a full tablet report...
I20260810 23:25:13.305702 17371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-2/data/instance:
uuid: "b2e15d7b4ac74741aca35eaa28e2a902"
format_stamp: "Formatted at 2026-08-10 23:25:13 on dist-test-slave-rmhg"
server_key: "8bf4073e65f5b0502063f79339a43c98"
server_key_iv: "a763c47a2ce54aadf54a09a7c3a4a231"
server_key_version: "encryptionkey@0"
I20260810 23:25:13.306725 17371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance:
uuid: "b2e15d7b4ac74741aca35eaa28e2a902"
format_stamp: "Formatted at 2026-08-10 23:25:13 on dist-test-slave-rmhg"
server_key: "8bf4073e65f5b0502063f79339a43c98"
server_key_iv: "a763c47a2ce54aadf54a09a7c3a4a231"
server_key_version: "encryptionkey@0"
I20260810 23:25:13.312880 17371 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260810 23:25:13.317001 17391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:13.318773 17371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:25:13.318966 17371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-2/wal
uuid: "b2e15d7b4ac74741aca35eaa28e2a902"
format_stamp: "Formatted at 2026-08-10 23:25:13 on dist-test-slave-rmhg"
server_key: "8bf4073e65f5b0502063f79339a43c98"
server_key_iv: "a763c47a2ce54aadf54a09a7c3a4a231"
server_key_version: "encryptionkey@0"
I20260810 23:25:13.319216 17371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:13.339497 17371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:13.344278 17371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:13.344576 17371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:13.347625 17371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:25:13.347805 17371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:13.347947 17371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:25:13.348017 17371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:25:13 dist-test-slave-rmhg krb5kdc[16715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404313, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:13.372227 16944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:13.348482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:47935 (local address 127.14.175.126:41753)
0810 23:25:13.348685 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:13.348691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:13.350448 (+ 1757us) server_negotiation.cc:407] Connection header received
0810 23:25:13.352755 (+ 2307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:13.352761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:13.352845 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:13.352934 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:13.355566 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:13.356623 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:13.359032 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:13.359450 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:13.359567 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:13.364169 (+ 4602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:13.364174 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:13.364177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:13.364215 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:13.367599 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:13.368607 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:13.368612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:13.368614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:13.368818 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:13.369545 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:13.369550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:13.369554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:13.369986 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:13.370147 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:13.371460 (+ 1313us) server_negotiation.cc:299] Negotiation successful
0810 23:25:13.371772 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 23:25:13.373095 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:13.349274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:41753 (local address 127.14.175.67:47935)
0810 23:25:13.350256 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:13.350306 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:13.352404 (+ 2098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:13.353263 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:13.353276 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:13.354287 (+ 1011us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:13.355295 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:13.355329 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:13.356825 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:13.356829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:13.358879 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:13.358892 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:13.359685 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:13.359694 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:13.359908 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:13.360684 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:13.360711 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:13.363896 (+ 3185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:13.367828 (+ 3932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:13.367845 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:13.367873 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:13.368454 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:13.368992 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:13.368998 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:13.369001 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:13.369400 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:13.370295 (+ 895us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:13.370304 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:13.370944 (+ 640us) client_negotiation.cc:769] Sending connection context
0810 23:25:13.371261 (+ 317us) client_negotiation.cc:240] Negotiation successful
0810 23:25:13.371769 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":445,"threads_started":1}
I20260810 23:25:13.394644 16944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:13.379216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:40545 (local address 127.14.175.126:41753)
0810 23:25:13.379430 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:13.379438 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:13.380072 (+ 634us) server_negotiation.cc:407] Connection header received
0810 23:25:13.380309 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:13.380315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:13.380415 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:13.380514 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:13.382124 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:13.383135 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:13.385223 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:13.385791 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:13.385922 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:13.388515 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:13.388520 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:13.388524 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:13.388557 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:13.391395 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:13.392377 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:13.392381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:13.392383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:13.392554 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:13.393030 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:13.393035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:13.393038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:13.393470 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:13.393617 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:13.394030 (+ 413us) server_negotiation.cc:299] Negotiation successful
0810 23:25:13.394279 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 23:25:13.394750 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:13.379188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:41753 (local address 127.14.175.67:40545)
0810 23:25:13.379945 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:13.379973 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:13.380152 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:13.380585 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:13.380591 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:13.381469 (+ 878us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:13.381962 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:13.381975 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:13.383310 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:13.383325 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:13.384950 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:13.384965 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:13.386173 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:13.386182 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:13.386375 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:13.387085 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:13.387111 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:13.388240 (+ 1129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:13.391576 (+ 3336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:13.391582 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:13.391586 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:13.392208 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:13.392714 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:13.392719 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:13.392723 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:13.392901 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:13.393592 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:13.393596 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:13.393794 (+ 198us) client_negotiation.cc:769] Sending connection context
0810 23:25:13.393916 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 23:25:13.394273 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":403,"threads_started":1}
I20260810 23:25:13.404495 17371 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.67:38159
I20260810 23:25:13.404495 17582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.67:38159 every 8 connection(s)
I20260810 23:25:13.405928 17371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
I20260810 23:25:13.408193 17586 heartbeater.cc:347] Connected to a master server at 127.14.175.126:41753
I20260810 23:25:13.408420 17586 heartbeater.cc:464] Registering TS with master...
I20260810 23:25:13.409015 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 17371
I20260810 23:25:13.409127 17586 heartbeater.cc:511] Master 127.14.175.126:41753 requested a full tablet report, sending...
I20260810 23:25:13.409188 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance
I20260810 23:25:13.409623 15037 external_mini_cluster.cc:1428] Setting key a1de2d144fdf9a7a0a49ddb9138e16b2
I20260810 23:25:13.410790 16790 ts_manager.cc:194] Registered new tserver with Master: b2e15d7b4ac74741aca35eaa28e2a902 (127.14.175.67:38159)
I20260810 23:25:13.411854 16790 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.67@KRBTEST.COM'} at 127.14.175.67:47935
I20260810 23:25:13.412858 15037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 23:25:13 dist-test-slave-rmhg krb5kdc[16715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404313, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 23:25:13 dist-test-slave-rmhg krb5kdc[16715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404313, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:13.464859 16944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:13.449651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47378 (local address 127.14.175.126:41753)
0810 23:25:13.449848 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:13.449853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:13.450043 (+ 190us) server_negotiation.cc:407] Connection header received
0810 23:25:13.450203 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:13.450207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:13.450281 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:13.450390 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:13.452006 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:13.452870 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:13.454591 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:13.455062 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:13.455185 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:13.459337 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:13.459344 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:13.459348 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:13.459394 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:13.462219 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:13.462953 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:13.462959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:13.462962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:13.463132 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:13.463439 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:13.463442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:13.463444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:13.463748 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:13.463944 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:13.464306 (+ 362us) server_negotiation.cc:299] Negotiation successful
0810 23:25:13.464528 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 23:25:13.469468 16791 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:25:13.477101 16791 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:25:13.525422 17609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:13.491400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:38183 (local address 127.0.0.1:48020)
0810 23:25:13.509680 (+ 18280us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:13.509704 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:13.509856 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:13.518154 (+ 8298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:13.518167 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:13.518210 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:25:13.518724 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:13.518739 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:13.520484 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:13.520490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:13.522094 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:13.522109 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:13.524328 (+ 2219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:13.524335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:13.524594 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:13.524670 (+ 76us) client_negotiation.cc:769] Sending connection context
0810 23:25:13.524829 (+ 159us) client_negotiation.cc:240] Negotiation successful
0810 23:25:13.525042 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18139,"thread_start_us":28486,"threads_started":1}
I20260810 23:25:13.526298 17611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:13.492026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48020 (local address 127.14.175.66:38183)
0810 23:25:13.516947 (+ 24921us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:13.516961 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:13.517005 (+ 44us) server_negotiation.cc:407] Connection header received
0810 23:25:13.517140 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:13.517175 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:13.517536 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:13.517808 (+ 272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:25:13.518956 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:13.520337 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:13.522285 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:13.523685 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:13.523829 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:13.524055 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:13.525909 (+ 1854us) server_negotiation.cc:299] Negotiation successful
0810 23:25:13.526064 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24672,"thread_start_us":25188,"threads_started":1}
I20260810 23:25:13.527410 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:13.491742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:46539 (local address 127.0.0.1:33988)
0810 23:25:13.518294 (+ 26552us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:13.518314 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:13.518458 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:13.519914 (+ 1456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:13.519918 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:13.519939 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:25:13.520483 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:13.520494 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:13.522459 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:13.522464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:13.524906 (+ 2442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:13.524918 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:13.526812 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:13.526817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:13.526944 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:13.526969 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 23:25:13.527043 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 23:25:13.527141 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26422,"mutex_wait_us":21,"thread_start_us":17630,"threads_started":1}
I20260810 23:25:13.527568 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:13.492399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33988 (local address 127.14.175.65:46539)
0810 23:25:13.517954 (+ 25555us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:13.517963 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:13.518436 (+ 473us) server_negotiation.cc:407] Connection header received
0810 23:25:13.519083 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:13.519093 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:13.519354 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:13.519579 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:25:13.520707 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:13.522322 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:13.525081 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:13.526615 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:13.526765 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:13.526970 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:13.527093 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 23:25:13.527264 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25401,"thread_start_us":25258,"threads_started":1}
I20260810 23:25:13.528543 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:13.491175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:38159 (local address 127.0.0.1:47144)
0810 23:25:13.506980 (+ 15805us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:13.507040 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:13.507298 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:13.519757 (+ 12459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:13.519762 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:13.519790 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:25:13.520384 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:13.520402 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:13.522693 (+ 2291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:13.522698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:13.524817 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:13.524834 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:13.527938 (+ 3104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:13.527945 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:13.528113 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:13.528137 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 23:25:13.528191 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 23:25:13.528286 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15416,"mutex_wait_us":9,"thread_start_us":15221,"threads_started":1}
I20260810 23:25:13.529456 17608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:13.491413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47144 (local address 127.14.175.67:38159)
0810 23:25:13.519122 (+ 27709us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:13.519130 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:13.519162 (+ 32us) server_negotiation.cc:407] Connection header received
0810 23:25:13.519246 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:13.519252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:13.519436 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:13.520597 (+ 1161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:25:13.520939 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:13.522564 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:13.525033 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:13.527682 (+ 2649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:13.528764 (+ 1082us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:13.528959 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:13.529026 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 23:25:13.529177 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27530,"thread_start_us":27345,"threads_started":1}
I20260810 23:25:13.530805 17235 tablet_service.cc:1467] Processing CreateTablet for tablet fe74f0adb1c34f44974f5aceac60ec9a (DEFAULT_TABLE table=test-table [id=0a79e42919d54553bee16e40a09563c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:25:13.531554 17000 tablet_service.cc:1467] Processing CreateTablet for tablet fe74f0adb1c34f44974f5aceac60ec9a (DEFAULT_TABLE table=test-table [id=0a79e42919d54553bee16e40a09563c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:25:13.532462 17235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe74f0adb1c34f44974f5aceac60ec9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:13.532986 17000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe74f0adb1c34f44974f5aceac60ec9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:13.532564 17487 tablet_service.cc:1467] Processing CreateTablet for tablet fe74f0adb1c34f44974f5aceac60ec9a (DEFAULT_TABLE table=test-table [id=0a79e42919d54553bee16e40a09563c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:25:13.533763 17487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe74f0adb1c34f44974f5aceac60ec9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:13.545737 17614 tablet_bootstrap.cc:492] T fe74f0adb1c34f44974f5aceac60ec9a P b2e15d7b4ac74741aca35eaa28e2a902: Bootstrap starting.
I20260810 23:25:13.549022 17615 tablet_bootstrap.cc:492] T fe74f0adb1c34f44974f5aceac60ec9a P d56e501300134723906fdfeb8637f97e: Bootstrap starting.
I20260810 23:25:13.550815 17614 tablet_bootstrap.cc:654] T fe74f0adb1c34f44974f5aceac60ec9a P b2e15d7b4ac74741aca35eaa28e2a902: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:13.550871 17616 tablet_bootstrap.cc:492] T fe74f0adb1c34f44974f5aceac60ec9a P aacd1a6c649b47e78f3a71c94a3f4437: Bootstrap starting.
I20260810 23:25:13.552438 17614 log.cc:824] T fe74f0adb1c34f44974f5aceac60ec9a P b2e15d7b4ac74741aca35eaa28e2a902: Log is configured to *not* fsync() on all Append() calls
I20260810 23:25:13.552966 17615 tablet_bootstrap.cc:654] T fe74f0adb1c34f44974f5aceac60ec9a P d56e501300134723906fdfeb8637f97e: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:13.554370 17615 log.cc:824] T fe74f0adb1c34f44974f5aceac60ec9a P d56e501300134723906fdfeb8637f97e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:25:13.554451 17616 tablet_bootstrap.cc:654] T fe74f0adb1c34f44974f5aceac60ec9a P aacd1a6c649b47e78f3a71c94a3f4437: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:13.555791 17616 log.cc:824] T fe74f0adb1c34f44974f5aceac60ec9a P aacd1a6c649b47e78f3a71c94a3f4437: Log is configured to *not* fsync() on all Append() calls
I20260810 23:25:13.556113 17614 tablet_bootstrap.cc:492] T fe74f0adb1c34f44974f5aceac60ec9a P b2e15d7b4ac74741aca35eaa28e2a902: No bootstrap required, opened a new log
I20260810 23:25:13.556428 17614 ts_tablet_manager.cc:1397] T fe74f0adb1c34f44974f5aceac60ec9a P b2e15d7b4ac74741aca35eaa28e2a902: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260810 23:25:13.557922 17615 tablet_bootstrap.cc:492] T fe74f0adb1c34f44974f5aceac60ec9a P d56e501300134723906fdfeb8637f97e: No bootstrap required, opened a new log
I20260810 23:25:13.558234 17615 ts_tablet_manager.cc:1397] T fe74f0adb1c34f44974f5aceac60ec9a P d56e501300134723906fdfeb8637f97e: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260810 23:25:13.559046 17616 tablet_bootstrap.cc:492] T fe74f0adb1c34f44974f5aceac60ec9a P aacd1a6c649b47e78f3a71c94a3f4437: No bootstrap required, opened a new log
I20260810 23:25:13.559402 17616 ts_tablet_manager.cc:1397] T fe74f0adb1c34f44974f5aceac60ec9a P aacd1a6c649b47e78f3a71c94a3f4437: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260810 23:25:13.566779 17616 raft_consensus.cc:348] T fe74f0adb1c34f44974f5aceac60ec9a P aacd1a6c649b47e78f3a71c94a3f4437 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } }
I20260810 23:25:13.567230 17616 raft_consensus.cc:374] T fe74f0adb1c34f44974f5aceac60ec9a P aacd1a6c649b47e78f3a71c94a3f4437 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:13.567341 17616 raft_consensus.cc:729] T fe74f0adb1c34f44974f5aceac60ec9a P aacd1a6c649b47e78f3a71c94a3f4437 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aacd1a6c649b47e78f3a71c94a3f4437, State: Initialized, Role: FOLLOWER
I20260810 23:25:13.567589 17614 raft_consensus.cc:348] T fe74f0adb1c34f44974f5aceac60ec9a P b2e15d7b4ac74741aca35eaa28e2a902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } }
I20260810 23:25:13.568084 17614 raft_consensus.cc:374] T fe74f0adb1c34f44974f5aceac60ec9a P b2e15d7b4ac74741aca35eaa28e2a902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:13.568187 17614 raft_consensus.cc:729] T fe74f0adb1c34f44974f5aceac60ec9a P b2e15d7b4ac74741aca35eaa28e2a902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2e15d7b4ac74741aca35eaa28e2a902, State: Initialized, Role: FOLLOWER
I20260810 23:25:13.568027 17616 consensus_queue.cc:260] T fe74f0adb1c34f44974f5aceac60ec9a P aacd1a6c649b47e78f3a71c94a3f4437 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } }
I20260810 23:25:13.568845 17614 consensus_queue.cc:260] T fe74f0adb1c34f44974f5aceac60ec9a P b2e15d7b4ac74741aca35eaa28e2a902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } }
I20260810 23:25:13.568995 17615 raft_consensus.cc:348] T fe74f0adb1c34f44974f5aceac60ec9a P d56e501300134723906fdfeb8637f97e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } }
I20260810 23:25:13.569484 17615 raft_consensus.cc:374] T fe74f0adb1c34f44974f5aceac60ec9a P d56e501300134723906fdfeb8637f97e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:13.569617 17615 raft_consensus.cc:729] T fe74f0adb1c34f44974f5aceac60ec9a P d56e501300134723906fdfeb8637f97e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d56e501300134723906fdfeb8637f97e, State: Initialized, Role: FOLLOWER
I20260810 23:25:13.570715 17586 heartbeater.cc:503] Master 127.14.175.126:41753 was elected leader, sending a full tablet report...
I20260810 23:25:13.571305 17614 ts_tablet_manager.cc:1428] T fe74f0adb1c34f44974f5aceac60ec9a P b2e15d7b4ac74741aca35eaa28e2a902: Time spent starting tablet: real 0.015s user 0.016s sys 0.001s
I20260810 23:25:13.570461 17615 consensus_queue.cc:260] T fe74f0adb1c34f44974f5aceac60ec9a P d56e501300134723906fdfeb8637f97e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } }
I20260810 23:25:13.574820 17616 ts_tablet_manager.cc:1428] T fe74f0adb1c34f44974f5aceac60ec9a P aacd1a6c649b47e78f3a71c94a3f4437: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260810 23:25:13.576058 17365 heartbeater.cc:503] Master 127.14.175.126:41753 was elected leader, sending a full tablet report...
I20260810 23:25:13.578060 17615 ts_tablet_manager.cc:1428] T fe74f0adb1c34f44974f5aceac60ec9a P d56e501300134723906fdfeb8637f97e: Time spent starting tablet: real 0.020s user 0.014s sys 0.004s
W20260810 23:25:13.607817 17366 tablet.cc:2369] T fe74f0adb1c34f44974f5aceac60ec9a P d56e501300134723906fdfeb8637f97e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:25:13.657675 17587 tablet.cc:2369] T fe74f0adb1c34f44974f5aceac60ec9a P b2e15d7b4ac74741aca35eaa28e2a902: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:25:13.795331 17120 tablet.cc:2369] T fe74f0adb1c34f44974f5aceac60ec9a P aacd1a6c649b47e78f3a71c94a3f4437: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:25:13.853718 17630 raft_consensus.cc:482] T fe74f0adb1c34f44974f5aceac60ec9a P d56e501300134723906fdfeb8637f97e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:25:13.854137 17630 raft_consensus.cc:504] T fe74f0adb1c34f44974f5aceac60ec9a P d56e501300134723906fdfeb8637f97e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } }
I20260810 23:25:13.856688 17630 leader_election.cc:288] T fe74f0adb1c34f44974f5aceac60ec9a P d56e501300134723906fdfeb8637f97e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2e15d7b4ac74741aca35eaa28e2a902 (127.14.175.67:38159), aacd1a6c649b47e78f3a71c94a3f4437 (127.14.175.65:46539)
I20260810 23:25:13.865108 17608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:13.857089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:37089 (local address 127.14.175.67:38159)
0810 23:25:13.857320 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:13.857327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:13.858580 (+ 1253us) server_negotiation.cc:407] Connection header received
0810 23:25:13.858872 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:13.858878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:13.858953 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:13.859024 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:25:13.859976 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:13.861002 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:13.863042 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:13.864094 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:13.864275 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:13.864354 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:13.864710 (+ 356us) server_negotiation.cc:299] Negotiation successful
0810 23:25:13.864845 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260810 23:25:13.865271 17634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:13.857649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:38159 (local address 127.14.175.66:37089)
0810 23:25:13.858467 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:13.858485 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:13.858695 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:13.859236 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:13.859240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:13.859276 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:25:13.859775 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:13.859789 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:13.861138 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:13.861142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:13.862863 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:13.862878 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:13.864328 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:13.864334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:13.864526 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:13.864564 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 23:25:13.864658 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 23:25:13.864798 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"mutex_wait_us":37,"thread_start_us":527,"threads_started":1}
I20260810 23:25:13.866168 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:13.857250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:59931 (local address 127.14.175.65:46539)
0810 23:25:13.857454 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:13.857460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:13.858327 (+ 867us) server_negotiation.cc:407] Connection header received
0810 23:25:13.858622 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:13.858634 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:13.858714 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:13.858807 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:25:13.859649 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:13.860676 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:13.863517 (+ 2841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:13.864643 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:13.864807 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:13.864873 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:13.865782 (+ 909us) server_negotiation.cc:299] Negotiation successful
0810 23:25:13.865928 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 23:25:13.866526 17631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:13.857329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:46539 (local address 127.14.175.66:59931)
0810 23:25:13.858184 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:13.858212 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:13.858461 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:13.859025 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:13.859030 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:13.859056 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:25:13.859497 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:13.859511 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:13.860842 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:13.860848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:13.863326 (+ 2478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:13.863343 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:13.865051 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:13.865057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:13.865612 (+ 555us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:13.865647 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 23:25:13.865738 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 23:25:13.865870 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":79,"thread_start_us":438,"threads_started":1}
I20260810 23:25:13.866379 17524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe74f0adb1c34f44974f5aceac60ec9a" candidate_uuid: "d56e501300134723906fdfeb8637f97e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" is_pre_election: true
I20260810 23:25:13.866981 17524 raft_consensus.cc:2454] T fe74f0adb1c34f44974f5aceac60ec9a P b2e15d7b4ac74741aca35eaa28e2a902 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d56e501300134723906fdfeb8637f97e in term 0.
I20260810 23:25:13.867660 17147 leader_election.cc:302] T fe74f0adb1c34f44974f5aceac60ec9a P d56e501300134723906fdfeb8637f97e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2e15d7b4ac74741aca35eaa28e2a902, d56e501300134723906fdfeb8637f97e; no voters:
I20260810 23:25:13.867589 17034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe74f0adb1c34f44974f5aceac60ec9a" candidate_uuid: "d56e501300134723906fdfeb8637f97e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" is_pre_election: true
I20260810 23:25:13.868069 17630 raft_consensus.cc:2781] T fe74f0adb1c34f44974f5aceac60ec9a P d56e501300134723906fdfeb8637f97e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:25:13.868094 17034 raft_consensus.cc:2454] T fe74f0adb1c34f44974f5aceac60ec9a P aacd1a6c649b47e78f3a71c94a3f4437 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d56e501300134723906fdfeb8637f97e in term 0.
I20260810 23:25:13.868219 17630 raft_consensus.cc:482] T fe74f0adb1c34f44974f5aceac60ec9a P d56e501300134723906fdfeb8637f97e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:25:13.868317 17630 raft_consensus.cc:3037] T fe74f0adb1c34f44974f5aceac60ec9a P d56e501300134723906fdfeb8637f97e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:13.871110 17630 raft_consensus.cc:504] T fe74f0adb1c34f44974f5aceac60ec9a P d56e501300134723906fdfeb8637f97e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } }
I20260810 23:25:13.871765 17630 leader_election.cc:288] T fe74f0adb1c34f44974f5aceac60ec9a P d56e501300134723906fdfeb8637f97e [CANDIDATE]: Term 1 election: Requested vote from peers b2e15d7b4ac74741aca35eaa28e2a902 (127.14.175.67:38159), aacd1a6c649b47e78f3a71c94a3f4437 (127.14.175.65:46539)
I20260810 23:25:13.872223 17524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe74f0adb1c34f44974f5aceac60ec9a" candidate_uuid: "d56e501300134723906fdfeb8637f97e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2e15d7b4ac74741aca35eaa28e2a902"
I20260810 23:25:13.872258 17034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe74f0adb1c34f44974f5aceac60ec9a" candidate_uuid: "d56e501300134723906fdfeb8637f97e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aacd1a6c649b47e78f3a71c94a3f4437"
I20260810 23:25:13.872466 17034 raft_consensus.cc:3037] T fe74f0adb1c34f44974f5aceac60ec9a P aacd1a6c649b47e78f3a71c94a3f4437 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:13.872447 17524 raft_consensus.cc:3037] T fe74f0adb1c34f44974f5aceac60ec9a P b2e15d7b4ac74741aca35eaa28e2a902 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:13.875067 17524 raft_consensus.cc:2454] T fe74f0adb1c34f44974f5aceac60ec9a P b2e15d7b4ac74741aca35eaa28e2a902 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d56e501300134723906fdfeb8637f97e in term 1.
I20260810 23:25:13.875223 17034 raft_consensus.cc:2454] T fe74f0adb1c34f44974f5aceac60ec9a P aacd1a6c649b47e78f3a71c94a3f4437 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d56e501300134723906fdfeb8637f97e in term 1.
I20260810 23:25:13.875597 17147 leader_election.cc:302] T fe74f0adb1c34f44974f5aceac60ec9a P d56e501300134723906fdfeb8637f97e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2e15d7b4ac74741aca35eaa28e2a902, d56e501300134723906fdfeb8637f97e; no voters:
I20260810 23:25:13.875976 17630 raft_consensus.cc:2781] T fe74f0adb1c34f44974f5aceac60ec9a P d56e501300134723906fdfeb8637f97e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:25:13.876439 17630 raft_consensus.cc:686] T fe74f0adb1c34f44974f5aceac60ec9a P d56e501300134723906fdfeb8637f97e [term 1 LEADER]: Becoming Leader. State: Replica: d56e501300134723906fdfeb8637f97e, State: Running, Role: LEADER
I20260810 23:25:13.876926 17630 consensus_queue.cc:237] T fe74f0adb1c34f44974f5aceac60ec9a P d56e501300134723906fdfeb8637f97e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } }
I20260810 23:25:13.881753 16790 catalog_manager.cc:5495] T fe74f0adb1c34f44974f5aceac60ec9a P d56e501300134723906fdfeb8637f97e reported cstate change: term changed from 0 to 1, leader changed from <none> to d56e501300134723906fdfeb8637f97e (127.14.175.66). New cstate: current_term: 1 leader_uuid: "d56e501300134723906fdfeb8637f97e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } health_report { overall_health: UNKNOWN } } }
I20260810 23:25:13.991818 17611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:13.985268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48032 (local address 127.14.175.66:38183)
0810 23:25:13.985475 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:13.985482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:13.986070 (+ 588us) server_negotiation.cc:407] Connection header received
0810 23:25:13.986439 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:13.986444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:13.986541 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:13.986671 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:25:13.987191 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:13.987922 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:13.989446 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:13.989864 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:13.989980 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:13.990498 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:25:13.991167 (+ 669us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:25:13.991317 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:13.991450 (+ 133us) server_negotiation.cc:299] Negotiation successful
0810 23:25:13.991543 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:25:14.001423 17034 raft_consensus.cc:1261] T fe74f0adb1c34f44974f5aceac60ec9a P aacd1a6c649b47e78f3a71c94a3f4437 [term 1 FOLLOWER]: Refusing update from remote peer d56e501300134723906fdfeb8637f97e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:25:14.001437 17524 raft_consensus.cc:1261] T fe74f0adb1c34f44974f5aceac60ec9a P b2e15d7b4ac74741aca35eaa28e2a902 [term 1 FOLLOWER]: Refusing update from remote peer d56e501300134723906fdfeb8637f97e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:25:14.002625 17635 consensus_queue.cc:1035] T fe74f0adb1c34f44974f5aceac60ec9a P d56e501300134723906fdfeb8637f97e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:25:14.003782 17635 consensus_queue.cc:1035] T fe74f0adb1c34f44974f5aceac60ec9a P d56e501300134723906fdfeb8637f97e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:25:14.041105 16790 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:25:14.041595 16790 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:25:14.046322 16790 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf20257f3724473b8ce824cb8fa6f987: Sending DeleteTablet for 3 replicas of tablet fe74f0adb1c34f44974f5aceac60ec9a
I20260810 23:25:14.048012 17487 tablet_service.cc:1514] Processing DeleteTablet for tablet fe74f0adb1c34f44974f5aceac60ec9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:25:14 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:47144
I20260810 23:25:14.048198 17000 tablet_service.cc:1514] Processing DeleteTablet for tablet fe74f0adb1c34f44974f5aceac60ec9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:25:14 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:33988
I20260810 23:25:14.048035 17235 tablet_service.cc:1514] Processing DeleteTablet for tablet fe74f0adb1c34f44974f5aceac60ec9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:25:14 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:48020
I20260810 23:25:14.052824 16790 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:25:14.054207 16790 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:25:14.063959 17658 tablet_replica.cc:330] T fe74f0adb1c34f44974f5aceac60ec9a P d56e501300134723906fdfeb8637f97e: stopping tablet replica
I20260810 23:25:14.064735 17658 raft_consensus.cc:2229] T fe74f0adb1c34f44974f5aceac60ec9a P d56e501300134723906fdfeb8637f97e [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:25:14.065552 17657 tablet_replica.cc:330] T fe74f0adb1c34f44974f5aceac60ec9a P aacd1a6c649b47e78f3a71c94a3f4437: stopping tablet replica
I20260810 23:25:14.066056 17657 raft_consensus.cc:2229] T fe74f0adb1c34f44974f5aceac60ec9a P aacd1a6c649b47e78f3a71c94a3f4437 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:25:14.066653 17657 raft_consensus.cc:2258] T fe74f0adb1c34f44974f5aceac60ec9a P aacd1a6c649b47e78f3a71c94a3f4437 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:25:14.067864 17658 raft_consensus.cc:2258] T fe74f0adb1c34f44974f5aceac60ec9a P d56e501300134723906fdfeb8637f97e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:25:14.068749 17481 tablet_service.cc:1467] Processing CreateTablet for tablet 8dbdd732c96541eb8c2dd55a72880960 (DEFAULT_TABLE table=test-table [id=d8d4c6a695d14268a732374e9783f70b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:25:14.069417 17481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dbdd732c96541eb8c2dd55a72880960. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:14.069715 17656 tablet_replica.cc:330] T fe74f0adb1c34f44974f5aceac60ec9a P b2e15d7b4ac74741aca35eaa28e2a902: stopping tablet replica
I20260810 23:25:14.070158 17656 raft_consensus.cc:2229] T fe74f0adb1c34f44974f5aceac60ec9a P b2e15d7b4ac74741aca35eaa28e2a902 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:25:14.070685 17657 ts_tablet_manager.cc:1905] T fe74f0adb1c34f44974f5aceac60ec9a P aacd1a6c649b47e78f3a71c94a3f4437: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:25:14.071920 17000 tablet_service.cc:1467] Processing CreateTablet for tablet 8dbdd732c96541eb8c2dd55a72880960 (DEFAULT_TABLE table=test-table [id=d8d4c6a695d14268a732374e9783f70b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:25:14.072433 17000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dbdd732c96541eb8c2dd55a72880960. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:14.072883 17658 ts_tablet_manager.cc:1905] T fe74f0adb1c34f44974f5aceac60ec9a P d56e501300134723906fdfeb8637f97e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:25:14.078946 17614 tablet_bootstrap.cc:492] T 8dbdd732c96541eb8c2dd55a72880960 P b2e15d7b4ac74741aca35eaa28e2a902: Bootstrap starting.
I20260810 23:25:14.079501 17235 tablet_service.cc:1467] Processing CreateTablet for tablet 8dbdd732c96541eb8c2dd55a72880960 (DEFAULT_TABLE table=test-table [id=d8d4c6a695d14268a732374e9783f70b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:25:14.080070 17235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dbdd732c96541eb8c2dd55a72880960. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:14.080724 17614 tablet_bootstrap.cc:654] T 8dbdd732c96541eb8c2dd55a72880960 P b2e15d7b4ac74741aca35eaa28e2a902: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:14.081095 17656 raft_consensus.cc:2258] T fe74f0adb1c34f44974f5aceac60ec9a P b2e15d7b4ac74741aca35eaa28e2a902 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:25:14.083647 17656 ts_tablet_manager.cc:1905] T fe74f0adb1c34f44974f5aceac60ec9a P b2e15d7b4ac74741aca35eaa28e2a902: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:25:14.086549 17657 ts_tablet_manager.cc:1918] T fe74f0adb1c34f44974f5aceac60ec9a P aacd1a6c649b47e78f3a71c94a3f4437: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:25:14.086864 17657 log.cc:1196] T fe74f0adb1c34f44974f5aceac60ec9a P aacd1a6c649b47e78f3a71c94a3f4437: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-0/wal/wals/fe74f0adb1c34f44974f5aceac60ec9a
I20260810 23:25:14.087453 17657 ts_tablet_manager.cc:1939] T fe74f0adb1c34f44974f5aceac60ec9a P aacd1a6c649b47e78f3a71c94a3f4437: Deleting consensus metadata
I20260810 23:25:14.089695 16770 catalog_manager.cc:4841] TS aacd1a6c649b47e78f3a71c94a3f4437 (127.14.175.65:46539): tablet fe74f0adb1c34f44974f5aceac60ec9a (table test-table [id=0a79e42919d54553bee16e40a09563c0]) successfully deleted
I20260810 23:25:14.091482 17656 ts_tablet_manager.cc:1918] T fe74f0adb1c34f44974f5aceac60ec9a P b2e15d7b4ac74741aca35eaa28e2a902: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:25:14.091662 17656 log.cc:1196] T fe74f0adb1c34f44974f5aceac60ec9a P b2e15d7b4ac74741aca35eaa28e2a902: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-2/wal/wals/fe74f0adb1c34f44974f5aceac60ec9a
I20260810 23:25:14.092238 17656 ts_tablet_manager.cc:1939] T fe74f0adb1c34f44974f5aceac60ec9a P b2e15d7b4ac74741aca35eaa28e2a902: Deleting consensus metadata
I20260810 23:25:14.095261 16772 catalog_manager.cc:4841] TS b2e15d7b4ac74741aca35eaa28e2a902 (127.14.175.67:38159): tablet fe74f0adb1c34f44974f5aceac60ec9a (table test-table [id=0a79e42919d54553bee16e40a09563c0]) successfully deleted
I20260810 23:25:14.097643 17658 ts_tablet_manager.cc:1918] T fe74f0adb1c34f44974f5aceac60ec9a P d56e501300134723906fdfeb8637f97e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:25:14.097848 17658 log.cc:1196] T fe74f0adb1c34f44974f5aceac60ec9a P d56e501300134723906fdfeb8637f97e: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-1/wal/wals/fe74f0adb1c34f44974f5aceac60ec9a
I20260810 23:25:14.098431 17658 ts_tablet_manager.cc:1939] T fe74f0adb1c34f44974f5aceac60ec9a P d56e501300134723906fdfeb8637f97e: Deleting consensus metadata
I20260810 23:25:14.099140 17614 tablet_bootstrap.cc:492] T 8dbdd732c96541eb8c2dd55a72880960 P b2e15d7b4ac74741aca35eaa28e2a902: No bootstrap required, opened a new log
I20260810 23:25:14.099339 17614 ts_tablet_manager.cc:1397] T 8dbdd732c96541eb8c2dd55a72880960 P b2e15d7b4ac74741aca35eaa28e2a902: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.010s
I20260810 23:25:14.100243 17614 raft_consensus.cc:348] T 8dbdd732c96541eb8c2dd55a72880960 P b2e15d7b4ac74741aca35eaa28e2a902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } }
I20260810 23:25:14.100411 16766 catalog_manager.cc:4841] TS d56e501300134723906fdfeb8637f97e (127.14.175.66:38183): tablet fe74f0adb1c34f44974f5aceac60ec9a (table test-table [id=0a79e42919d54553bee16e40a09563c0]) successfully deleted
I20260810 23:25:14.100502 17614 raft_consensus.cc:374] T 8dbdd732c96541eb8c2dd55a72880960 P b2e15d7b4ac74741aca35eaa28e2a902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:14.100579 17614 raft_consensus.cc:729] T 8dbdd732c96541eb8c2dd55a72880960 P b2e15d7b4ac74741aca35eaa28e2a902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2e15d7b4ac74741aca35eaa28e2a902, State: Initialized, Role: FOLLOWER
I20260810 23:25:14.100967 17614 consensus_queue.cc:260] T 8dbdd732c96541eb8c2dd55a72880960 P b2e15d7b4ac74741aca35eaa28e2a902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } }
I20260810 23:25:14.104307 17614 ts_tablet_manager.cc:1428] T 8dbdd732c96541eb8c2dd55a72880960 P b2e15d7b4ac74741aca35eaa28e2a902: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260810 23:25:14.107801 17661 tablet_bootstrap.cc:492] T 8dbdd732c96541eb8c2dd55a72880960 P aacd1a6c649b47e78f3a71c94a3f4437: Bootstrap starting.
I20260810 23:25:14.108372 17663 tablet_bootstrap.cc:492] T 8dbdd732c96541eb8c2dd55a72880960 P d56e501300134723906fdfeb8637f97e: Bootstrap starting.
I20260810 23:25:14.110540 17661 tablet_bootstrap.cc:654] T 8dbdd732c96541eb8c2dd55a72880960 P aacd1a6c649b47e78f3a71c94a3f4437: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:14.110543 17663 tablet_bootstrap.cc:654] T 8dbdd732c96541eb8c2dd55a72880960 P d56e501300134723906fdfeb8637f97e: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:14.113108 17661 tablet_bootstrap.cc:492] T 8dbdd732c96541eb8c2dd55a72880960 P aacd1a6c649b47e78f3a71c94a3f4437: No bootstrap required, opened a new log
I20260810 23:25:14.113319 17661 ts_tablet_manager.cc:1397] T 8dbdd732c96541eb8c2dd55a72880960 P aacd1a6c649b47e78f3a71c94a3f4437: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:25:14.113454 17663 tablet_bootstrap.cc:492] T 8dbdd732c96541eb8c2dd55a72880960 P d56e501300134723906fdfeb8637f97e: No bootstrap required, opened a new log
I20260810 23:25:14.113626 17663 ts_tablet_manager.cc:1397] T 8dbdd732c96541eb8c2dd55a72880960 P d56e501300134723906fdfeb8637f97e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20260810 23:25:14.114090 17661 raft_consensus.cc:348] T 8dbdd732c96541eb8c2dd55a72880960 P aacd1a6c649b47e78f3a71c94a3f4437 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } }
I20260810 23:25:14.114301 17661 raft_consensus.cc:374] T 8dbdd732c96541eb8c2dd55a72880960 P aacd1a6c649b47e78f3a71c94a3f4437 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:14.114396 17661 raft_consensus.cc:729] T 8dbdd732c96541eb8c2dd55a72880960 P aacd1a6c649b47e78f3a71c94a3f4437 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aacd1a6c649b47e78f3a71c94a3f4437, State: Initialized, Role: FOLLOWER
I20260810 23:25:14.114637 17661 consensus_queue.cc:260] T 8dbdd732c96541eb8c2dd55a72880960 P aacd1a6c649b47e78f3a71c94a3f4437 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } }
I20260810 23:25:14.114554 17663 raft_consensus.cc:348] T 8dbdd732c96541eb8c2dd55a72880960 P d56e501300134723906fdfeb8637f97e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } }
I20260810 23:25:14.114837 17663 raft_consensus.cc:374] T 8dbdd732c96541eb8c2dd55a72880960 P d56e501300134723906fdfeb8637f97e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:14.114954 17663 raft_consensus.cc:729] T 8dbdd732c96541eb8c2dd55a72880960 P d56e501300134723906fdfeb8637f97e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d56e501300134723906fdfeb8637f97e, State: Initialized, Role: FOLLOWER
I20260810 23:25:14.115175 17661 ts_tablet_manager.cc:1428] T 8dbdd732c96541eb8c2dd55a72880960 P aacd1a6c649b47e78f3a71c94a3f4437: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 23:25:14.115304 17663 consensus_queue.cc:260] T 8dbdd732c96541eb8c2dd55a72880960 P d56e501300134723906fdfeb8637f97e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } }
I20260810 23:25:14.115896 17663 ts_tablet_manager.cc:1428] T 8dbdd732c96541eb8c2dd55a72880960 P d56e501300134723906fdfeb8637f97e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 23:25:14.285308 17628 raft_consensus.cc:482] T 8dbdd732c96541eb8c2dd55a72880960 P aacd1a6c649b47e78f3a71c94a3f4437 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:25:14.285601 17628 raft_consensus.cc:504] T 8dbdd732c96541eb8c2dd55a72880960 P aacd1a6c649b47e78f3a71c94a3f4437 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } }
I20260810 23:25:14.287772 17628 leader_election.cc:288] T 8dbdd732c96541eb8c2dd55a72880960 P aacd1a6c649b47e78f3a71c94a3f4437 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d56e501300134723906fdfeb8637f97e (127.14.175.66:38183), b2e15d7b4ac74741aca35eaa28e2a902 (127.14.175.67:38159)
I20260810 23:25:14.298164 17611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:14.288260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:46549 (local address 127.14.175.66:38183)
0810 23:25:14.288486 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:14.288494 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:14.289304 (+ 810us) server_negotiation.cc:407] Connection header received
0810 23:25:14.289620 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:14.289626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:14.289702 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:14.289788 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:25:14.290696 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:14.291828 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:14.295209 (+ 3381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:14.296716 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:14.296856 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:14.296933 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:14.297585 (+ 652us) server_negotiation.cc:299] Negotiation successful
0810 23:25:14.297817 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 23:25:14.299170 17668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:14.288089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:38183 (local address 127.14.175.65:46549)
0810 23:25:14.289187 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:14.289210 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:14.289437 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:14.290086 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:14.290091 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:14.290124 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:25:14.290495 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:14.290506 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:14.292505 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:14.292510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:14.294979 (+ 2469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:14.294995 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:14.297129 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:14.297136 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:14.297377 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:14.297414 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 23:25:14.298651 (+ 1237us) client_negotiation.cc:240] Negotiation successful
0810 23:25:14.298813 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":949,"thread_start_us":752,"threads_started":1}
I20260810 23:25:14.300222 17279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dbdd732c96541eb8c2dd55a72880960" candidate_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d56e501300134723906fdfeb8637f97e" is_pre_election: true
I20260810 23:25:14.300719 17279 raft_consensus.cc:2454] T 8dbdd732c96541eb8c2dd55a72880960 P d56e501300134723906fdfeb8637f97e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aacd1a6c649b47e78f3a71c94a3f4437 in term 0.
I20260810 23:25:14.301542 16933 leader_election.cc:302] T 8dbdd732c96541eb8c2dd55a72880960 P aacd1a6c649b47e78f3a71c94a3f4437 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aacd1a6c649b47e78f3a71c94a3f4437, d56e501300134723906fdfeb8637f97e; no voters:
I20260810 23:25:14.304196 17628 raft_consensus.cc:2781] T 8dbdd732c96541eb8c2dd55a72880960 P aacd1a6c649b47e78f3a71c94a3f4437 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:25:14.304440 17628 raft_consensus.cc:482] T 8dbdd732c96541eb8c2dd55a72880960 P aacd1a6c649b47e78f3a71c94a3f4437 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:25:14.304656 17628 raft_consensus.cc:3037] T 8dbdd732c96541eb8c2dd55a72880960 P aacd1a6c649b47e78f3a71c94a3f4437 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:14.308382 17628 raft_consensus.cc:504] T 8dbdd732c96541eb8c2dd55a72880960 P aacd1a6c649b47e78f3a71c94a3f4437 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } }
I20260810 23:25:14.309795 17608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:14.288296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:33891 (local address 127.14.175.67:38159)
0810 23:25:14.288658 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:14.288665 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:14.302328 (+ 13663us) server_negotiation.cc:407] Connection header received
0810 23:25:14.302600 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:14.302606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:14.302711 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:14.302779 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:25:14.303373 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:14.304567 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:14.307158 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:14.308472 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:14.308605 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:14.308688 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:14.309360 (+ 672us) server_negotiation.cc:299] Negotiation successful
0810 23:25:14.309516 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223}
I20260810 23:25:14.310542 17668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:14.288531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:38159 (local address 127.14.175.65:33891)
0810 23:25:14.302226 (+ 13695us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:14.302246 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:14.302438 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:14.302973 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:14.302978 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:14.303016 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:25:14.303191 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:14.303203 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:14.304779 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:14.304784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:14.306926 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:14.306944 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:14.308875 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:14.308907 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:14.309169 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:14.309204 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 23:25:14.309991 (+ 787us) client_negotiation.cc:240] Negotiation successful
0810 23:25:14.310175 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13582,"thread_start_us":20160,"threads_started":1}
I20260810 23:25:14.311600 17524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dbdd732c96541eb8c2dd55a72880960" candidate_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" is_pre_election: true
I20260810 23:25:14.311854 17524 raft_consensus.cc:2454] T 8dbdd732c96541eb8c2dd55a72880960 P b2e15d7b4ac74741aca35eaa28e2a902 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aacd1a6c649b47e78f3a71c94a3f4437 in term 0.
I20260810 23:25:14.313690 17279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dbdd732c96541eb8c2dd55a72880960" candidate_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d56e501300134723906fdfeb8637f97e"
I20260810 23:25:14.313865 17628 leader_election.cc:288] T 8dbdd732c96541eb8c2dd55a72880960 P aacd1a6c649b47e78f3a71c94a3f4437 [CANDIDATE]: Term 1 election: Requested vote from peers d56e501300134723906fdfeb8637f97e (127.14.175.66:38183), b2e15d7b4ac74741aca35eaa28e2a902 (127.14.175.67:38159)
I20260810 23:25:14.313942 17279 raft_consensus.cc:3037] T 8dbdd732c96541eb8c2dd55a72880960 P d56e501300134723906fdfeb8637f97e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:14.314683 17524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dbdd732c96541eb8c2dd55a72880960" candidate_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2e15d7b4ac74741aca35eaa28e2a902"
I20260810 23:25:14.314915 17524 raft_consensus.cc:3037] T 8dbdd732c96541eb8c2dd55a72880960 P b2e15d7b4ac74741aca35eaa28e2a902 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:14.317364 17279 raft_consensus.cc:2454] T 8dbdd732c96541eb8c2dd55a72880960 P d56e501300134723906fdfeb8637f97e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aacd1a6c649b47e78f3a71c94a3f4437 in term 1.
I20260810 23:25:14.317792 17524 raft_consensus.cc:2454] T 8dbdd732c96541eb8c2dd55a72880960 P b2e15d7b4ac74741aca35eaa28e2a902 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aacd1a6c649b47e78f3a71c94a3f4437 in term 1.
I20260810 23:25:14.317857 16933 leader_election.cc:302] T 8dbdd732c96541eb8c2dd55a72880960 P aacd1a6c649b47e78f3a71c94a3f4437 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aacd1a6c649b47e78f3a71c94a3f4437, d56e501300134723906fdfeb8637f97e; no voters:
I20260810 23:25:14.318243 17628 raft_consensus.cc:2781] T 8dbdd732c96541eb8c2dd55a72880960 P aacd1a6c649b47e78f3a71c94a3f4437 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:25:14.318732 17628 raft_consensus.cc:686] T 8dbdd732c96541eb8c2dd55a72880960 P aacd1a6c649b47e78f3a71c94a3f4437 [term 1 LEADER]: Becoming Leader. State: Replica: aacd1a6c649b47e78f3a71c94a3f4437, State: Running, Role: LEADER
I20260810 23:25:14.319473 17628 consensus_queue.cc:237] T 8dbdd732c96541eb8c2dd55a72880960 P aacd1a6c649b47e78f3a71c94a3f4437 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } }
I20260810 23:25:14.323626 16791 catalog_manager.cc:5495] T 8dbdd732c96541eb8c2dd55a72880960 P aacd1a6c649b47e78f3a71c94a3f4437 reported cstate change: term changed from 0 to 1, leader changed from <none> to aacd1a6c649b47e78f3a71c94a3f4437 (127.14.175.65). New cstate: current_term: 1 leader_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } health_report { overall_health: UNKNOWN } } }
Aug 10 23:25:14 dist-test-slave-rmhg krb5kdc[16715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404311, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:14.441519 17684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:14.407937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:41753 (local address 127.0.0.1:47380)
0810 23:25:14.423945 (+ 16008us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:14.423967 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:14.424141 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:14.424641 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:14.424647 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:14.425615 (+ 968us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:14.425974 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:14.425989 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:14.427553 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:14.427559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:14.429128 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:14.429145 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:14.430169 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:14.430176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:14.430425 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:14.431307 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:14.431337 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:14.434243 (+ 2906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:14.438638 (+ 4395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:14.438655 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:14.438662 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:14.439145 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:14.439624 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:14.439643 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:14.439645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:14.439797 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:14.440557 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:14.440565 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:14.440768 (+ 203us) client_negotiation.cc:769] Sending connection context
0810 23:25:14.440899 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 23:25:14.441130 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15873,"thread_start_us":15728,"threads_started":1}
I20260810 23:25:14.441875 17683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:14.407958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47380 (local address 127.14.175.126:41753)
0810 23:25:14.423492 (+ 15534us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:14.423500 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:14.424231 (+ 731us) server_negotiation.cc:407] Connection header received
0810 23:25:14.424311 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:14.424317 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:14.424407 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:14.424572 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:14.426190 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:14.427383 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:14.429317 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:14.429955 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:14.430117 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:14.434536 (+ 4419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:14.434544 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:14.434549 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:14.434592 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:14.438457 (+ 3865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:14.439309 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:14.439315 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:14.439319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:14.439497 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:14.439936 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:14.439942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:14.439945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:14.440403 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:14.440566 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:14.441100 (+ 534us) server_negotiation.cc:299] Negotiation successful
0810 23:25:14.441390 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15362,"thread_start_us":16146,"threads_started":1}
I20260810 23:25:14.448348 16791 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:47380:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 23:25:14.450150 16791 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:25:14.458099 17000 tablet_service.cc:1467] Processing CreateTablet for tablet 9c5ec572820f482eb26b431ac23463c5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=06d8eda7c2d04380a10f4e4bcf319c33]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:25:14.458616 17481 tablet_service.cc:1467] Processing CreateTablet for tablet 9c5ec572820f482eb26b431ac23463c5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=06d8eda7c2d04380a10f4e4bcf319c33]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:25:14.458622 17235 tablet_service.cc:1467] Processing CreateTablet for tablet 9c5ec572820f482eb26b431ac23463c5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=06d8eda7c2d04380a10f4e4bcf319c33]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:25:14.458858 17000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c5ec572820f482eb26b431ac23463c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:14.459177 17481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c5ec572820f482eb26b431ac23463c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:14.459211 17235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c5ec572820f482eb26b431ac23463c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:14.466952 17663 tablet_bootstrap.cc:492] T 9c5ec572820f482eb26b431ac23463c5 P d56e501300134723906fdfeb8637f97e: Bootstrap starting.
I20260810 23:25:14.467065 17661 tablet_bootstrap.cc:492] T 9c5ec572820f482eb26b431ac23463c5 P aacd1a6c649b47e78f3a71c94a3f4437: Bootstrap starting.
I20260810 23:25:14.467640 17614 tablet_bootstrap.cc:492] T 9c5ec572820f482eb26b431ac23463c5 P b2e15d7b4ac74741aca35eaa28e2a902: Bootstrap starting.
I20260810 23:25:14.468945 17661 tablet_bootstrap.cc:654] T 9c5ec572820f482eb26b431ac23463c5 P aacd1a6c649b47e78f3a71c94a3f4437: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:14.468942 17663 tablet_bootstrap.cc:654] T 9c5ec572820f482eb26b431ac23463c5 P d56e501300134723906fdfeb8637f97e: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:14.469540 17614 tablet_bootstrap.cc:654] T 9c5ec572820f482eb26b431ac23463c5 P b2e15d7b4ac74741aca35eaa28e2a902: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:14.478083 17614 tablet_bootstrap.cc:492] T 9c5ec572820f482eb26b431ac23463c5 P b2e15d7b4ac74741aca35eaa28e2a902: No bootstrap required, opened a new log
I20260810 23:25:14.478327 17614 ts_tablet_manager.cc:1397] T 9c5ec572820f482eb26b431ac23463c5 P b2e15d7b4ac74741aca35eaa28e2a902: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260810 23:25:14.478824 17663 tablet_bootstrap.cc:492] T 9c5ec572820f482eb26b431ac23463c5 P d56e501300134723906fdfeb8637f97e: No bootstrap required, opened a new log
I20260810 23:25:14.479099 17663 ts_tablet_manager.cc:1397] T 9c5ec572820f482eb26b431ac23463c5 P d56e501300134723906fdfeb8637f97e: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20260810 23:25:14.479211 17614 raft_consensus.cc:348] T 9c5ec572820f482eb26b431ac23463c5 P b2e15d7b4ac74741aca35eaa28e2a902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } }
I20260810 23:25:14.479482 17614 raft_consensus.cc:374] T 9c5ec572820f482eb26b431ac23463c5 P b2e15d7b4ac74741aca35eaa28e2a902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:14.479578 17614 raft_consensus.cc:729] T 9c5ec572820f482eb26b431ac23463c5 P b2e15d7b4ac74741aca35eaa28e2a902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2e15d7b4ac74741aca35eaa28e2a902, State: Initialized, Role: FOLLOWER
I20260810 23:25:14.480142 17663 raft_consensus.cc:348] T 9c5ec572820f482eb26b431ac23463c5 P d56e501300134723906fdfeb8637f97e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } }
I20260810 23:25:14.480562 17663 raft_consensus.cc:374] T 9c5ec572820f482eb26b431ac23463c5 P d56e501300134723906fdfeb8637f97e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:14.480639 17661 tablet_bootstrap.cc:492] T 9c5ec572820f482eb26b431ac23463c5 P aacd1a6c649b47e78f3a71c94a3f4437: No bootstrap required, opened a new log
I20260810 23:25:14.480700 17663 raft_consensus.cc:729] T 9c5ec572820f482eb26b431ac23463c5 P d56e501300134723906fdfeb8637f97e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d56e501300134723906fdfeb8637f97e, State: Initialized, Role: FOLLOWER
I20260810 23:25:14.480904 17661 ts_tablet_manager.cc:1397] T 9c5ec572820f482eb26b431ac23463c5 P aacd1a6c649b47e78f3a71c94a3f4437: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260810 23:25:14.481240 17614 consensus_queue.cc:260] T 9c5ec572820f482eb26b431ac23463c5 P b2e15d7b4ac74741aca35eaa28e2a902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } }
I20260810 23:25:14.481264 17663 consensus_queue.cc:260] T 9c5ec572820f482eb26b431ac23463c5 P d56e501300134723906fdfeb8637f97e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } }
I20260810 23:25:14.481846 17661 raft_consensus.cc:348] T 9c5ec572820f482eb26b431ac23463c5 P aacd1a6c649b47e78f3a71c94a3f4437 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } }
I20260810 23:25:14.481861 17630 tablet_replica.cc:439] T 9c5ec572820f482eb26b431ac23463c5 P d56e501300134723906fdfeb8637f97e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } } }
I20260810 23:25:14.481890 17629 tablet_replica.cc:439] T 9c5ec572820f482eb26b431ac23463c5 P b2e15d7b4ac74741aca35eaa28e2a902: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } } }
I20260810 23:25:14.482183 17630 tablet_replica.cc:442] T 9c5ec572820f482eb26b431ac23463c5 P d56e501300134723906fdfeb8637f97e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:25:14.482179 17661 raft_consensus.cc:374] T 9c5ec572820f482eb26b431ac23463c5 P aacd1a6c649b47e78f3a71c94a3f4437 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:14.482286 17629 tablet_replica.cc:442] T 9c5ec572820f482eb26b431ac23463c5 P b2e15d7b4ac74741aca35eaa28e2a902: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:25:14.482304 17661 raft_consensus.cc:729] T 9c5ec572820f482eb26b431ac23463c5 P aacd1a6c649b47e78f3a71c94a3f4437 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aacd1a6c649b47e78f3a71c94a3f4437, State: Initialized, Role: FOLLOWER
I20260810 23:25:14.482637 17661 consensus_queue.cc:260] T 9c5ec572820f482eb26b431ac23463c5 P aacd1a6c649b47e78f3a71c94a3f4437 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } }
I20260810 23:25:14.483273 17628 tablet_replica.cc:439] T 9c5ec572820f482eb26b431ac23463c5 P aacd1a6c649b47e78f3a71c94a3f4437: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } } }
I20260810 23:25:14.483587 17628 tablet_replica.cc:442] T 9c5ec572820f482eb26b431ac23463c5 P aacd1a6c649b47e78f3a71c94a3f4437: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:25:14.483741 17614 ts_tablet_manager.cc:1428] T 9c5ec572820f482eb26b431ac23463c5 P b2e15d7b4ac74741aca35eaa28e2a902: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260810 23:25:14.485425 17663 ts_tablet_manager.cc:1428] T 9c5ec572820f482eb26b431ac23463c5 P d56e501300134723906fdfeb8637f97e: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260810 23:25:14.483279 17661 ts_tablet_manager.cc:1428] T 9c5ec572820f482eb26b431ac23463c5 P aacd1a6c649b47e78f3a71c94a3f4437: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 23:25:14.488300 17628 raft_consensus.cc:482] T 9c5ec572820f482eb26b431ac23463c5 P aacd1a6c649b47e78f3a71c94a3f4437 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:25:14.488469 17628 raft_consensus.cc:504] T 9c5ec572820f482eb26b431ac23463c5 P aacd1a6c649b47e78f3a71c94a3f4437 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } }
I20260810 23:25:14.489099 17628 leader_election.cc:288] T 9c5ec572820f482eb26b431ac23463c5 P aacd1a6c649b47e78f3a71c94a3f4437 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d56e501300134723906fdfeb8637f97e (127.14.175.66:38183), b2e15d7b4ac74741aca35eaa28e2a902 (127.14.175.67:38159)
I20260810 23:25:14.489568 17524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c5ec572820f482eb26b431ac23463c5" candidate_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" is_pre_election: true
I20260810 23:25:14.489588 17279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c5ec572820f482eb26b431ac23463c5" candidate_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d56e501300134723906fdfeb8637f97e" is_pre_election: true
I20260810 23:25:14.489852 17279 raft_consensus.cc:2454] T 9c5ec572820f482eb26b431ac23463c5 P d56e501300134723906fdfeb8637f97e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aacd1a6c649b47e78f3a71c94a3f4437 in term 0.
I20260810 23:25:14.489846 17524 raft_consensus.cc:2454] T 9c5ec572820f482eb26b431ac23463c5 P b2e15d7b4ac74741aca35eaa28e2a902 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aacd1a6c649b47e78f3a71c94a3f4437 in term 0.
I20260810 23:25:14.490288 16937 leader_election.cc:302] T 9c5ec572820f482eb26b431ac23463c5 P aacd1a6c649b47e78f3a71c94a3f4437 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aacd1a6c649b47e78f3a71c94a3f4437, b2e15d7b4ac74741aca35eaa28e2a902; no voters:
I20260810 23:25:14.490638 17628 raft_consensus.cc:2781] T 9c5ec572820f482eb26b431ac23463c5 P aacd1a6c649b47e78f3a71c94a3f4437 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:25:14.490751 17628 raft_consensus.cc:482] T 9c5ec572820f482eb26b431ac23463c5 P aacd1a6c649b47e78f3a71c94a3f4437 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:25:14.490855 17628 raft_consensus.cc:3037] T 9c5ec572820f482eb26b431ac23463c5 P aacd1a6c649b47e78f3a71c94a3f4437 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:14.493904 17628 raft_consensus.cc:504] T 9c5ec572820f482eb26b431ac23463c5 P aacd1a6c649b47e78f3a71c94a3f4437 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } }
I20260810 23:25:14.494694 17628 leader_election.cc:288] T 9c5ec572820f482eb26b431ac23463c5 P aacd1a6c649b47e78f3a71c94a3f4437 [CANDIDATE]: Term 1 election: Requested vote from peers d56e501300134723906fdfeb8637f97e (127.14.175.66:38183), b2e15d7b4ac74741aca35eaa28e2a902 (127.14.175.67:38159)
I20260810 23:25:14.495289 17524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c5ec572820f482eb26b431ac23463c5" candidate_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2e15d7b4ac74741aca35eaa28e2a902"
I20260810 23:25:14.495551 17524 raft_consensus.cc:3037] T 9c5ec572820f482eb26b431ac23463c5 P b2e15d7b4ac74741aca35eaa28e2a902 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:14.498454 17524 raft_consensus.cc:2454] T 9c5ec572820f482eb26b431ac23463c5 P b2e15d7b4ac74741aca35eaa28e2a902 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aacd1a6c649b47e78f3a71c94a3f4437 in term 1.
I20260810 23:25:14.498910 16937 leader_election.cc:302] T 9c5ec572820f482eb26b431ac23463c5 P aacd1a6c649b47e78f3a71c94a3f4437 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aacd1a6c649b47e78f3a71c94a3f4437, b2e15d7b4ac74741aca35eaa28e2a902; no voters:
I20260810 23:25:14.498930 17279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c5ec572820f482eb26b431ac23463c5" candidate_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d56e501300134723906fdfeb8637f97e"
I20260810 23:25:14.499197 17279 raft_consensus.cc:3037] T 9c5ec572820f482eb26b431ac23463c5 P d56e501300134723906fdfeb8637f97e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:14.499228 17628 raft_consensus.cc:2781] T 9c5ec572820f482eb26b431ac23463c5 P aacd1a6c649b47e78f3a71c94a3f4437 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:25:14.499379 17628 raft_consensus.cc:686] T 9c5ec572820f482eb26b431ac23463c5 P aacd1a6c649b47e78f3a71c94a3f4437 [term 1 LEADER]: Becoming Leader. State: Replica: aacd1a6c649b47e78f3a71c94a3f4437, State: Running, Role: LEADER
I20260810 23:25:14.499634 17628 consensus_queue.cc:237] T 9c5ec572820f482eb26b431ac23463c5 P aacd1a6c649b47e78f3a71c94a3f4437 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } }
I20260810 23:25:14.501478 17670 tablet_replica.cc:439] T 9c5ec572820f482eb26b431ac23463c5 P aacd1a6c649b47e78f3a71c94a3f4437: TxnStatusTablet state changed. Reason: New leader aacd1a6c649b47e78f3a71c94a3f4437. Latest consensus state: current_term: 1 leader_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } } }
I20260810 23:25:14.501767 17670 tablet_replica.cc:442] T 9c5ec572820f482eb26b431ac23463c5 P aacd1a6c649b47e78f3a71c94a3f4437: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:25:14.502431 17279 raft_consensus.cc:2454] T 9c5ec572820f482eb26b431ac23463c5 P d56e501300134723906fdfeb8637f97e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aacd1a6c649b47e78f3a71c94a3f4437 in term 1.
I20260810 23:25:14.504310 16786 catalog_manager.cc:5495] T 9c5ec572820f482eb26b431ac23463c5 P aacd1a6c649b47e78f3a71c94a3f4437 reported cstate change: term changed from 0 to 1, leader changed from <none> to aacd1a6c649b47e78f3a71c94a3f4437 (127.14.175.65). New cstate: current_term: 1 leader_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } health_report { overall_health: UNKNOWN } } }
I20260810 23:25:14.505856 17691 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 23:25:14.506357 17691 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 23:25:14.537194 17684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:14.522949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:46539 (local address 127.0.0.1:33996)
0810 23:25:14.523382 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:14.523403 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:14.523557 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:14.530259 (+ 6702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:14.530265 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:14.530282 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:25:14.530501 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:14.530514 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:14.532019 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:14.532025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:14.533724 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:14.533746 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:14.534837 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:14.534845 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:14.535093 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:14.535204 (+ 111us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:25:14.536531 (+ 1327us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:25:14.536565 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 23:25:14.536656 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 23:25:14.536829 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239}
I20260810 23:25:14.537236 17695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:14.523210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33996 (local address 127.14.175.65:46539)
0810 23:25:14.529683 (+ 6473us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:14.529690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:14.529717 (+ 27us) server_negotiation.cc:407] Connection header received
0810 23:25:14.529835 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:14.529840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:14.529967 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:14.530174 (+ 207us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:25:14.530713 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:14.531852 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:14.533932 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:14.534646 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:14.534775 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:14.535482 (+ 707us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:25:14.536394 (+ 912us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:25:14.536549 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:14.536704 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 23:25:14.536805 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6312,"thread_start_us":6140,"threads_started":1}
I20260810 23:25:14.543375 17524 raft_consensus.cc:1261] T 9c5ec572820f482eb26b431ac23463c5 P b2e15d7b4ac74741aca35eaa28e2a902 [term 1 FOLLOWER]: Refusing update from remote peer aacd1a6c649b47e78f3a71c94a3f4437: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:25:14.543771 17279 raft_consensus.cc:1261] T 9c5ec572820f482eb26b431ac23463c5 P d56e501300134723906fdfeb8637f97e [term 1 FOLLOWER]: Refusing update from remote peer aacd1a6c649b47e78f3a71c94a3f4437: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:25:14.544284 17670 consensus_queue.cc:1035] T 9c5ec572820f482eb26b431ac23463c5 P aacd1a6c649b47e78f3a71c94a3f4437 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:25:14.544723 17628 consensus_queue.cc:1035] T 9c5ec572820f482eb26b431ac23463c5 P aacd1a6c649b47e78f3a71c94a3f4437 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:25:14.552217 17629 tablet_replica.cc:439] T 9c5ec572820f482eb26b431ac23463c5 P b2e15d7b4ac74741aca35eaa28e2a902: TxnStatusTablet state changed. Reason: New leader aacd1a6c649b47e78f3a71c94a3f4437. Latest consensus state: current_term: 1 leader_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } } }
I20260810 23:25:14.552578 17629 tablet_replica.cc:442] T 9c5ec572820f482eb26b431ac23463c5 P b2e15d7b4ac74741aca35eaa28e2a902: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:25:14.557658 17629 tablet_replica.cc:439] T 9c5ec572820f482eb26b431ac23463c5 P b2e15d7b4ac74741aca35eaa28e2a902: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } } }
I20260810 23:25:14.557989 17629 tablet_replica.cc:442] T 9c5ec572820f482eb26b431ac23463c5 P b2e15d7b4ac74741aca35eaa28e2a902: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:25:14.561749 17670 tablet_replica.cc:439] T 9c5ec572820f482eb26b431ac23463c5 P aacd1a6c649b47e78f3a71c94a3f4437: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } } }
I20260810 23:25:14.562108 17670 tablet_replica.cc:442] T 9c5ec572820f482eb26b431ac23463c5 P aacd1a6c649b47e78f3a71c94a3f4437: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:25:14.564033 17630 tablet_replica.cc:439] T 9c5ec572820f482eb26b431ac23463c5 P d56e501300134723906fdfeb8637f97e: TxnStatusTablet state changed. Reason: New leader aacd1a6c649b47e78f3a71c94a3f4437. Latest consensus state: current_term: 1 leader_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } } }
I20260810 23:25:14.564399 17630 tablet_replica.cc:442] T 9c5ec572820f482eb26b431ac23463c5 P d56e501300134723906fdfeb8637f97e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:25:14.573802 17628 tablet_replica.cc:439] T 9c5ec572820f482eb26b431ac23463c5 P aacd1a6c649b47e78f3a71c94a3f4437: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } } }
I20260810 23:25:14.574349 17628 tablet_replica.cc:442] T 9c5ec572820f482eb26b431ac23463c5 P aacd1a6c649b47e78f3a71c94a3f4437: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:25:14.580289 17630 tablet_replica.cc:439] T 9c5ec572820f482eb26b431ac23463c5 P d56e501300134723906fdfeb8637f97e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aacd1a6c649b47e78f3a71c94a3f4437" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46539 } } peers { permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 } } peers { permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 } } }
I20260810 23:25:14.580607 17630 tablet_replica.cc:442] T 9c5ec572820f482eb26b431ac23463c5 P d56e501300134723906fdfeb8637f97e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:25:14.585912 17695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:14.578479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34006 (local address 127.14.175.65:46539)
0810 23:25:14.578678 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:14.578685 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:14.579315 (+ 630us) server_negotiation.cc:407] Connection header received
0810 23:25:14.579636 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:14.579642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:14.579739 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:14.579808 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:25:14.580742 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:14.581946 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:14.583720 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:14.584424 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:14.584595 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:14.585017 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:25:14.585207 (+ 190us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:25:14.585299 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:14.585495 (+ 196us) server_negotiation.cc:299] Negotiation successful
0810 23:25:14.585588 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 23:25:14.605538 17695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:14.598366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:47591 (local address 127.14.175.65:46539)
0810 23:25:14.598527 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:14.598532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:14.599218 (+ 686us) server_negotiation.cc:407] Connection header received
0810 23:25:14.599405 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:14.599411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:14.599512 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:14.599621 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:25:14.600129 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:14.601337 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:14.603162 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:14.603913 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:14.604052 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:14.604556 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:25:14.604721 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:25:14.604814 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:14.604977 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 23:25:14.605070 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 23:25:14.605615 17712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:14.598445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:46539 (local address 127.14.175.65:47591)
0810 23:25:14.599115 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:14.599132 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:14.599267 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:14.599638 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:14.599641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:14.599650 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:25:14.599966 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:14.599978 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:14.601578 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:14.601582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:14.602986 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:14.603005 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:14.604143 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:14.604148 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:14.604283 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:14.604351 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:25:14.604834 (+ 483us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:25:14.604849 (+ 15us) client_negotiation.cc:769] Sending connection context
0810 23:25:14.604928 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 23:25:14.605054 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":356,"threads_started":1}
I20260810 23:25:14.617556 17279 raft_consensus.cc:1261] T 8dbdd732c96541eb8c2dd55a72880960 P d56e501300134723906fdfeb8637f97e [term 1 FOLLOWER]: Refusing update from remote peer aacd1a6c649b47e78f3a71c94a3f4437: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:25:14.617609 17524 raft_consensus.cc:1261] T 8dbdd732c96541eb8c2dd55a72880960 P b2e15d7b4ac74741aca35eaa28e2a902 [term 1 FOLLOWER]: Refusing update from remote peer aacd1a6c649b47e78f3a71c94a3f4437: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:25:14.618157 17670 consensus_queue.cc:1035] T 8dbdd732c96541eb8c2dd55a72880960 P aacd1a6c649b47e78f3a71c94a3f4437 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d56e501300134723906fdfeb8637f97e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 38183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:25:14.618468 17705 consensus_queue.cc:1035] T 8dbdd732c96541eb8c2dd55a72880960 P aacd1a6c649b47e78f3a71c94a3f4437 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e15d7b4ac74741aca35eaa28e2a902" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 38159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:25:14.635963 17697 mvcc.cc:205] Tried to move back new op lower bound from 7317112072667918336 to 7317112071457456128. Current Snapshot: MvccSnapshot[applied={T|T < 7317112072667918336}]
I20260810 23:25:14.651856 17698 mvcc.cc:205] Tried to move back new op lower bound from 7317112072667918336 to 7317112071457456128. Current Snapshot: MvccSnapshot[applied={T|T < 7317112072667918336}]
I20260810 23:25:14.722492 16788 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:25:14.722774 16788 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:25:14.726428 16788 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf20257f3724473b8ce824cb8fa6f987: Sending DeleteTablet for 3 replicas of tablet 8dbdd732c96541eb8c2dd55a72880960
I20260810 23:25:14.727392 17235 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dbdd732c96541eb8c2dd55a72880960 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:25:14 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:48020
I20260810 23:25:14.727420 16998 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dbdd732c96541eb8c2dd55a72880960 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:25:14 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:33988
I20260810 23:25:14.727885 17481 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dbdd732c96541eb8c2dd55a72880960 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:25:14 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:47144
I20260810 23:25:14.742833 17724 tablet_replica.cc:330] T 8dbdd732c96541eb8c2dd55a72880960 P b2e15d7b4ac74741aca35eaa28e2a902: stopping tablet replica
I20260810 23:25:14.743085 17722 tablet_replica.cc:330] T 8dbdd732c96541eb8c2dd55a72880960 P d56e501300134723906fdfeb8637f97e: stopping tablet replica
I20260810 23:25:14.743294 17724 raft_consensus.cc:2229] T 8dbdd732c96541eb8c2dd55a72880960 P b2e15d7b4ac74741aca35eaa28e2a902 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:25:14.743323 17722 raft_consensus.cc:2229] T 8dbdd732c96541eb8c2dd55a72880960 P d56e501300134723906fdfeb8637f97e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:25:14.743628 17722 raft_consensus.cc:2258] T 8dbdd732c96541eb8c2dd55a72880960 P d56e501300134723906fdfeb8637f97e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:25:14.743687 17724 raft_consensus.cc:2258] T 8dbdd732c96541eb8c2dd55a72880960 P b2e15d7b4ac74741aca35eaa28e2a902 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:25:14.745008 17724 ts_tablet_manager.cc:1905] T 8dbdd732c96541eb8c2dd55a72880960 P b2e15d7b4ac74741aca35eaa28e2a902: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:25:14.745060 17722 ts_tablet_manager.cc:1905] T 8dbdd732c96541eb8c2dd55a72880960 P d56e501300134723906fdfeb8637f97e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:25:14.753248 17724 ts_tablet_manager.cc:1918] T 8dbdd732c96541eb8c2dd55a72880960 P b2e15d7b4ac74741aca35eaa28e2a902: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:25:14.753958 17723 tablet_replica.cc:330] T 8dbdd732c96541eb8c2dd55a72880960 P aacd1a6c649b47e78f3a71c94a3f4437: stopping tablet replica
I20260810 23:25:14.754276 17723 raft_consensus.cc:2229] T 8dbdd732c96541eb8c2dd55a72880960 P aacd1a6c649b47e78f3a71c94a3f4437 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:25:14.754614 17722 ts_tablet_manager.cc:1918] T 8dbdd732c96541eb8c2dd55a72880960 P d56e501300134723906fdfeb8637f97e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:25:14.754776 17722 log.cc:1196] T 8dbdd732c96541eb8c2dd55a72880960 P d56e501300134723906fdfeb8637f97e: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-1/wal/wals/8dbdd732c96541eb8c2dd55a72880960
I20260810 23:25:14.755266 17723 raft_consensus.cc:2258] T 8dbdd732c96541eb8c2dd55a72880960 P aacd1a6c649b47e78f3a71c94a3f4437 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:25:14.755308 17722 ts_tablet_manager.cc:1939] T 8dbdd732c96541eb8c2dd55a72880960 P d56e501300134723906fdfeb8637f97e: Deleting consensus metadata
I20260810 23:25:14.756578 16766 catalog_manager.cc:4841] TS d56e501300134723906fdfeb8637f97e (127.14.175.66:38183): tablet 8dbdd732c96541eb8c2dd55a72880960 (table test-table [id=d8d4c6a695d14268a732374e9783f70b]) successfully deleted
I20260810 23:25:14.756932 17723 ts_tablet_manager.cc:1905] T 8dbdd732c96541eb8c2dd55a72880960 P aacd1a6c649b47e78f3a71c94a3f4437: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:25:14.757167 17724 log.cc:1196] T 8dbdd732c96541eb8c2dd55a72880960 P b2e15d7b4ac74741aca35eaa28e2a902: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-2/wal/wals/8dbdd732c96541eb8c2dd55a72880960
I20260810 23:25:14.757691 17724 ts_tablet_manager.cc:1939] T 8dbdd732c96541eb8c2dd55a72880960 P b2e15d7b4ac74741aca35eaa28e2a902: Deleting consensus metadata
I20260810 23:25:14.758913 16772 catalog_manager.cc:4841] TS b2e15d7b4ac74741aca35eaa28e2a902 (127.14.175.67:38159): tablet 8dbdd732c96541eb8c2dd55a72880960 (table test-table [id=d8d4c6a695d14268a732374e9783f70b]) successfully deleted
Aug 10 23:25:14 dist-test-slave-rmhg krb5kdc[16715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404313, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.175.65@KRBTEST.COM
I20260810 23:25:14.766403 17723 ts_tablet_manager.cc:1918] T 8dbdd732c96541eb8c2dd55a72880960 P aacd1a6c649b47e78f3a71c94a3f4437: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:25:14.766590 17723 log.cc:1196] T 8dbdd732c96541eb8c2dd55a72880960 P aacd1a6c649b47e78f3a71c94a3f4437: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404299514076-15037-0/minicluster-data/ts-0/wal/wals/8dbdd732c96541eb8c2dd55a72880960
I20260810 23:25:14.767145 17723 ts_tablet_manager.cc:1939] T 8dbdd732c96541eb8c2dd55a72880960 P aacd1a6c649b47e78f3a71c94a3f4437: Deleting consensus metadata
I20260810 23:25:14.767961 17695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:14.736585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34008 (local address 127.14.175.65:46539)
0810 23:25:14.736833 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:14.736840 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:14.744008 (+ 7168us) server_negotiation.cc:407] Connection header received
0810 23:25:14.745439 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:14.745446 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:14.745540 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:14.745618 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:14.749192 (+ 3574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:14.750381 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:14.754236 (+ 3855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:14.755129 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:14.755265 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:14.760884 (+ 5619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:14.760900 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:14.760918 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:14.760958 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:14.764431 (+ 3473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:14.765352 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:14.765360 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:14.765363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:14.765572 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:14.765997 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:14.766002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:14.766004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:14.766409 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:14.766966 (+ 557us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:14.767174 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 23:25:14.767652 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 23:25:14.768450 16770 catalog_manager.cc:4841] TS aacd1a6c649b47e78f3a71c94a3f4437 (127.14.175.65:46539): tablet 8dbdd732c96541eb8c2dd55a72880960 (table test-table [id=d8d4c6a695d14268a732374e9783f70b]) successfully deleted
W20260810 23:25:14.768877 17113 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34008
I20260810 23:25:14.788708 17683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:14.772286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47394 (local address 127.14.175.126:41753)
0810 23:25:14.772540 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:14.772547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:14.773690 (+ 1143us) server_negotiation.cc:407] Connection header received
0810 23:25:14.773880 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:14.773887 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:14.773998 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:14.774110 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:14.775671 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:14.776873 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:14.778256 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:14.778917 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:14.779078 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:14.781500 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:14.781507 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:14.781512 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:14.781559 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:14.785378 (+ 3819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:14.786245 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:14.786252 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:14.786256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:14.786434 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:14.786960 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:14.786966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:14.786969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:14.787406 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:14.787686 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:14.788117 (+ 431us) server_negotiation.cc:299] Negotiation successful
0810 23:25:14.788376 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
W20260810 23:25:14.789417 16788 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47394
I20260810 23:25:14.791610 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 16901
I20260810 23:25:14.809240 17695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:14.792786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34010 (local address 127.14.175.65:46539)
0810 23:25:14.793055 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:14.793061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:14.793082 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:25:14.793170 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:14.793175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:14.793255 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:14.793342 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:14.794729 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:14.795596 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:14.796828 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:14.797440 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:14.797590 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:14.802892 (+ 5302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:14.802897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:14.802900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:14.802942 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:14.806188 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:14.806935 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:14.806940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:14.806942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:14.807114 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:14.807583 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:14.807587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:14.807590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:14.807920 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:14.808192 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:14.808585 (+ 393us) server_negotiation.cc:299] Negotiation successful
0810 23:25:14.808837 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260810 23:25:14.809811 17113 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34010
W20260810 23:25:14.810382 15037 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 16901: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:25:14.810654 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 16901
I20260810 23:25:14.846038 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 17123
Aug 10 23:25:14 dist-test-slave-rmhg krb5kdc[16715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404313, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.175.66@KRBTEST.COM
I20260810 23:25:14.862903 17734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:14.846898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48034 (local address 127.14.175.66:38183)
0810 23:25:14.847386 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:14.847390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:14.847408 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:25:14.847465 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:14.847468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:14.847536 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:14.847637 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:14.848862 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:14.849688 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:14.850984 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:14.851376 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:14.851468 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:14.855449 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:14.855470 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:14.855503 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:14.855557 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:14.859625 (+ 4068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:14.860476 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:14.860487 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:14.860491 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:14.860689 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:14.861220 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:14.861225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:14.861228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:14.861719 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:14.862042 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:14.862270 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 23:25:14.862529 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":251,"threads_started":1}
W20260810 23:25:14.863708 17357 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48034
W20260810 23:25:14.864435 15037 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 17123: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:25:14.864619 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 17123
I20260810 23:25:14.897933 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 17371
Aug 10 23:25:14 dist-test-slave-rmhg krb5kdc[16715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404313, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.175.67@KRBTEST.COM
I20260810 23:25:14.914631 17735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:14.898847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47152 (local address 127.14.175.67:38159)
0810 23:25:14.899405 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:14.899410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:14.899425 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:25:14.899481 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:14.899485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:14.899557 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:14.899664 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:14.901040 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:14.901857 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:14.903206 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:14.903686 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:14.903804 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:14.908339 (+ 4535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:14.908359 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:14.908379 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:14.908428 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:14.911472 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:14.912330 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:14.912340 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:14.912343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:14.912508 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:14.912925 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:14.912930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:14.912932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:14.913422 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:14.913721 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:14.914056 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 23:25:14.914304 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":335,"threads_started":1}
W20260810 23:25:14.915385 17578 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47152
W20260810 23:25:14.916182 15037 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 17371: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:25:14.916369 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 17371
I20260810 23:25:14.943815 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 16740
I20260810 23:25:14.945047 16851 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:25:15.063633 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 16740
2026-08-10T23:25:15Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5361 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:25:15 dist-test-slave-rmhg krb5kdc[17740](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:25:15 dist-test-slave-rmhg krb5kdc[17740](info): set up 2 sockets
Aug 10 23:25:15 dist-test-slave-rmhg krb5kdc[17740](info): commencing operation
krb5kdc: starting...
W20260810 23:25:16.293026 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.174s user 0.006s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:25:16 dist-test-slave-rmhg krb5kdc[17740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:25:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:25:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
I20260810 23:25:16.513145 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:32813
--webserver_interface=127.14.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:32773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:32813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:16.867165 17764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:16.867494 17764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:16.867638 17764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:16.875162 17764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:25:16.875283 17764 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 23:25:16.875424 17764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:16.875569 17764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:16.875636 17764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:25:16.875723 17764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:25:16.887005 17764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:32773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:32813
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:32813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:16.888499 17764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:16.890050 17764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:16.903419 17772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:16.903293 17773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:16.903447 17776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:16.904112 17764 server_base.cc:1034] running on GCE node
I20260810 23:25:16.905058 17764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:16.906687 17764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:16.907955 17764 hybrid_clock.cc:648] HybridClock initialized: now 1786404316907858 us; error 74 us; skew 500 ppm
I20260810 23:25:16.908321 17764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:16 dist-test-slave-rmhg krb5kdc[17740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404316, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:16.911814 17764 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:25:16.913507 17764 webserver.cc:460] Webserver started at http://127.14.175.126:38429/ using document root <none> and password file <none>
I20260810 23:25:16.914253 17764 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:16.914368 17764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:16.914710 17764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:16.918020 17764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "8420095ccbb64e43b12d94041d536850"
format_stamp: "Formatted at 2026-08-10 23:25:16 on dist-test-slave-rmhg"
server_key: "2539a2bd7647b168c928050f5f1fe9fc"
server_key_iv: "22090671c39213a92206d52b7d6d0528"
server_key_version: "encryptionkey@0"
I20260810 23:25:16.918861 17764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "8420095ccbb64e43b12d94041d536850"
format_stamp: "Formatted at 2026-08-10 23:25:16 on dist-test-slave-rmhg"
server_key: "2539a2bd7647b168c928050f5f1fe9fc"
server_key_iv: "22090671c39213a92206d52b7d6d0528"
server_key_version: "encryptionkey@0"
I20260810 23:25:16.924579 17764 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:25:16.928467 17787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:16.930055 17764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:25:16.930256 17764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "8420095ccbb64e43b12d94041d536850"
format_stamp: "Formatted at 2026-08-10 23:25:16 on dist-test-slave-rmhg"
server_key: "2539a2bd7647b168c928050f5f1fe9fc"
server_key_iv: "22090671c39213a92206d52b7d6d0528"
server_key_version: "encryptionkey@0"
I20260810 23:25:16.930488 17764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:16.969492 17764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:16.974182 17764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:16.974462 17764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:16.996835 17764 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:32813
I20260810 23:25:16.996857 17888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:32813 every 8 connection(s)
I20260810 23:25:16.998425 17764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:25:17.002564 17892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:17.005831 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 17764
I20260810 23:25:17.006131 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:25:17.006927 15037 external_mini_cluster.cc:1428] Setting key 0f1388975c6d9b42e3022f257535c3d6
I20260810 23:25:17.016332 17892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8420095ccbb64e43b12d94041d536850: Bootstrap starting.
I20260810 23:25:17.020459 17892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8420095ccbb64e43b12d94041d536850: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:17.021945 17892 log.cc:824] T 00000000000000000000000000000000 P 8420095ccbb64e43b12d94041d536850: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:25:17 dist-test-slave-rmhg krb5kdc[17740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:17.025556 17892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8420095ccbb64e43b12d94041d536850: No bootstrap required, opened a new log
I20260810 23:25:17.032395 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:17.009328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47400 (local address 127.14.175.126:32813)
0810 23:25:17.010238 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:17.010251 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:17.010297 (+ 46us) server_negotiation.cc:407] Connection header received
0810 23:25:17.012068 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:17.012096 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:17.012558 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:17.012925 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:17.014849 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:17.016294 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:17.018201 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:17.018967 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:17.019195 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:17.024407 (+ 5212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:17.024415 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:17.024433 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:17.024496 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:17.027780 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:17.028697 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:17.028708 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:17.028716 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:17.028901 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:17.029358 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:17.029362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:17.029364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:17.029996 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:17.030255 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:17.030724 (+ 469us) server_negotiation.cc:299] Negotiation successful
0810 23:25:17.031149 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":434,"threads_started":1}
I20260810 23:25:17.035055 17892 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8420095ccbb64e43b12d94041d536850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8420095ccbb64e43b12d94041d536850" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 32813 } }
I20260810 23:25:17.035534 17892 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8420095ccbb64e43b12d94041d536850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:17.035658 17892 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8420095ccbb64e43b12d94041d536850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8420095ccbb64e43b12d94041d536850, State: Initialized, Role: FOLLOWER
I20260810 23:25:17.036357 17892 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8420095ccbb64e43b12d94041d536850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8420095ccbb64e43b12d94041d536850" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 32813 } }
I20260810 23:25:17.036621 17892 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8420095ccbb64e43b12d94041d536850 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:25:17.036751 17892 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8420095ccbb64e43b12d94041d536850 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:25:17.036911 17892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8420095ccbb64e43b12d94041d536850 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:17.039284 17892 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8420095ccbb64e43b12d94041d536850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8420095ccbb64e43b12d94041d536850" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 32813 } }
I20260810 23:25:17.039861 17892 leader_election.cc:302] T 00000000000000000000000000000000 P 8420095ccbb64e43b12d94041d536850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8420095ccbb64e43b12d94041d536850; no voters:
I20260810 23:25:17.040604 17892 leader_election.cc:288] T 00000000000000000000000000000000 P 8420095ccbb64e43b12d94041d536850 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:25:17.040794 17904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8420095ccbb64e43b12d94041d536850 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:25:17.041595 17904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8420095ccbb64e43b12d94041d536850 [term 1 LEADER]: Becoming Leader. State: Replica: 8420095ccbb64e43b12d94041d536850, State: Running, Role: LEADER
I20260810 23:25:17.042151 17904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8420095ccbb64e43b12d94041d536850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8420095ccbb64e43b12d94041d536850" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 32813 } }
I20260810 23:25:17.042933 17892 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8420095ccbb64e43b12d94041d536850 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:25:17.045217 17906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8420095ccbb64e43b12d94041d536850 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8420095ccbb64e43b12d94041d536850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8420095ccbb64e43b12d94041d536850" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 32813 } } }
I20260810 23:25:17.045674 17906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8420095ccbb64e43b12d94041d536850 [sys.catalog]: This master's current role is: LEADER
I20260810 23:25:17.046376 17907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8420095ccbb64e43b12d94041d536850 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8420095ccbb64e43b12d94041d536850. Latest consensus state: current_term: 1 leader_uuid: "8420095ccbb64e43b12d94041d536850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8420095ccbb64e43b12d94041d536850" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 32813 } } }
I20260810 23:25:17.046608 17916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:25:17.046672 17907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8420095ccbb64e43b12d94041d536850 [sys.catalog]: This master's current role is: LEADER
I20260810 23:25:17.053238 17916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:25:17.062749 17916 catalog_manager.cc:1347] Generated new cluster ID: c3a9771054c6410bbf602aa9e7c5e1c9
I20260810 23:25:17.062894 17916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:25:17.092270 17916 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:25:17.093976 17916 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:25:17.100014 17916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8420095ccbb64e43b12d94041d536850: Generated new TSK 0
I20260810 23:25:17.100898 17916 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
I20260810 23:25:17.161480 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.65:0
--local_ip_for_outbound_sockets=127.14.175.65
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:32813
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:32773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:17.534469 17939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:17.534857 17939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:17.535029 17939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:17.543011 17939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:17.543169 17939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:17.543291 17939 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 23:25:17.543352 17939 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:25:17.543458 17939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.65
I20260810 23:25:17.554199 17939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:32773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.175.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.175.126:32813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.65
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:17.555763 17939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:17.557228 17939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:17.571554 17946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:17.571537 17948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:17.571661 17951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:17.572458 17939 server_base.cc:1034] running on GCE node
I20260810 23:25:17.573252 17939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:17.574332 17939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:17.575562 17939 hybrid_clock.cc:648] HybridClock initialized: now 1786404317575505 us; error 52 us; skew 500 ppm
I20260810 23:25:17.576020 17939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:17 dist-test-slave-rmhg krb5kdc[17740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404317, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:17.579970 17939 init.cc:377] Logged in from keytab as kudu/127.14.175.65@KRBTEST.COM (short username kudu)
I20260810 23:25:17.581919 17939 webserver.cc:460] Webserver started at http://127.14.175.65:46679/ using document root <none> and password file <none>
I20260810 23:25:17.582722 17939 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:17.582857 17939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:17.583242 17939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:17.586797 17939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-0/data/instance:
uuid: "deadca232e58457395f6519bdc10494e"
format_stamp: "Formatted at 2026-08-10 23:25:17 on dist-test-slave-rmhg"
server_key: "930a7cbf2f718bc3792254cb03314822"
server_key_iv: "22171bcd96cf8994fd70d0d5eb79c070"
server_key_version: "encryptionkey@0"
I20260810 23:25:17.587679 17939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance:
uuid: "deadca232e58457395f6519bdc10494e"
format_stamp: "Formatted at 2026-08-10 23:25:17 on dist-test-slave-rmhg"
server_key: "930a7cbf2f718bc3792254cb03314822"
server_key_iv: "22171bcd96cf8994fd70d0d5eb79c070"
server_key_version: "encryptionkey@0"
I20260810 23:25:17.594113 17939 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260810 23:25:17.598580 17959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:17.600493 17939 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260810 23:25:17.600761 17939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-0/wal
uuid: "deadca232e58457395f6519bdc10494e"
format_stamp: "Formatted at 2026-08-10 23:25:17 on dist-test-slave-rmhg"
server_key: "930a7cbf2f718bc3792254cb03314822"
server_key_iv: "22171bcd96cf8994fd70d0d5eb79c070"
server_key_version: "encryptionkey@0"
I20260810 23:25:17.600998 17939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:17.629419 17939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:17.634539 17939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:17.634872 17939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:17.638084 17939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:25:17.638212 17939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:17.638337 17939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:25:17.638456 17939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:25:17 dist-test-slave-rmhg krb5kdc[17740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404317, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:17.662539 17969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:17.638698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:44341 (local address 127.14.175.126:32813)
0810 23:25:17.639385 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:17.639392 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:17.640785 (+ 1393us) server_negotiation.cc:407] Connection header received
0810 23:25:17.643021 (+ 2236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:17.643027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:17.643162 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:17.643277 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:17.645383 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:17.646645 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:17.649512 (+ 2867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:17.649995 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:17.650137 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:17.655303 (+ 5166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:17.655308 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:17.655312 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:17.655351 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:17.658119 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:17.659324 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:17.659329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:17.659332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:17.659495 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:17.660222 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:17.660229 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:17.660233 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:17.660616 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:17.660770 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:17.661841 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0810 23:25:17.662119 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":433,"threads_started":1}
I20260810 23:25:17.664145 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:17.639438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:32813 (local address 127.14.175.65:44341)
0810 23:25:17.640512 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:17.640566 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:17.642680 (+ 2114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:17.643544 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:17.643554 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:17.644146 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:17.645111 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:17.645172 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:17.646946 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:17.646951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:17.649299 (+ 2348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:17.649328 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:17.650310 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:17.650323 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:17.650744 (+ 421us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:17.651694 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:17.651727 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:17.655006 (+ 3279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:17.658321 (+ 3315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:17.658334 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:17.658359 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:17.659108 (+ 749us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:17.659650 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:17.659656 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:17.659660 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:17.660037 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:17.660764 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:17.660773 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:17.661321 (+ 548us) client_negotiation.cc:769] Sending connection context
0810 23:25:17.661697 (+ 376us) client_negotiation.cc:240] Negotiation successful
0810 23:25:17.662356 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":359,"threads_started":1}
I20260810 23:25:17.686890 18078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:17.671252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:32813 (local address 127.14.175.65:51445)
0810 23:25:17.671896 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:17.671920 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:17.672118 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:17.672537 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:17.672541 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:17.673345 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:17.673818 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:17.673831 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:17.675430 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:17.675436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:17.676523 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:17.676539 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:17.677662 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:17.677670 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:17.677899 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:17.678709 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:17.678741 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:17.680085 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:17.683926 (+ 3841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:17.683933 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:17.683937 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:17.684590 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:17.685006 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:17.685011 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:17.685014 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:17.685198 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:17.685926 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:17.685931 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:17.686127 (+ 196us) client_negotiation.cc:769] Sending connection context
0810 23:25:17.686245 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 23:25:17.686532 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":336,"threads_started":1}
I20260810 23:25:17.687116 17969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:17.671315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:51445 (local address 127.14.175.126:32813)
0810 23:25:17.671546 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:17.671553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:17.672016 (+ 463us) server_negotiation.cc:407] Connection header received
0810 23:25:17.672273 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:17.672279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:17.672372 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:17.672501 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:17.674089 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:17.675259 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:17.676686 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:17.677420 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:17.677578 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:17.680326 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:17.680332 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:17.680335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:17.680371 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:17.683725 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:17.684728 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:17.684735 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:17.684739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:17.684888 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:17.685313 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:17.685319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:17.685322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:17.685688 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:17.685865 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:17.686425 (+ 560us) server_negotiation.cc:299] Negotiation successful
0810 23:25:17.686724 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 23:25:17.692147 17939 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.65:44481
I20260810 23:25:17.692227 18146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.65:44481 every 8 connection(s)
I20260810 23:25:17.693504 17939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
I20260810 23:25:17.695796 18150 heartbeater.cc:347] Connected to a master server at 127.14.175.126:32813
I20260810 23:25:17.696025 18150 heartbeater.cc:464] Registering TS with master...
I20260810 23:25:17.696702 18150 heartbeater.cc:511] Master 127.14.175.126:32813 requested a full tablet report, sending...
I20260810 23:25:17.698163 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 17939
I20260810 23:25:17.698424 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance
I20260810 23:25:17.699273 15037 external_mini_cluster.cc:1428] Setting key b9205695055ba1e953087ee1291b6208
I20260810 23:25:17.699390 17819 ts_manager.cc:194] Registered new tserver with Master: deadca232e58457395f6519bdc10494e (127.14.175.65:44481)
I20260810 23:25:17.701297 17819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.65@KRBTEST.COM'} at 127.14.175.65:44341
WARNING: no policy specified for kudu/127.14.175.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
I20260810 23:25:17.748991 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.66:0
--local_ip_for_outbound_sockets=127.14.175.66
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:32813
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:32773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:18.095279 18155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:18.095628 18155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:18.095780 18155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:18.103798 18155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:18.103962 18155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:18.104085 18155 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 23:25:18.104156 18155 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:25:18.104262 18155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.66
I20260810 23:25:18.115121 18155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:32773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.175.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.175.126:32813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.66
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:18.116570 18155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:18.118109 18155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:18.133033 18161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:18.133134 18163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:18.133103 18160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:18.133847 18155 server_base.cc:1034] running on GCE node
I20260810 23:25:18.134578 18155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:18.135645 18155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:18.136893 18155 hybrid_clock.cc:648] HybridClock initialized: now 1786404318136811 us; error 69 us; skew 500 ppm
I20260810 23:25:18.137305 18155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:18 dist-test-slave-rmhg krb5kdc[17740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404318, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:18.140656 18155 init.cc:377] Logged in from keytab as kudu/127.14.175.66@KRBTEST.COM (short username kudu)
I20260810 23:25:18.142376 18155 webserver.cc:460] Webserver started at http://127.14.175.66:39469/ using document root <none> and password file <none>
I20260810 23:25:18.143159 18155 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:18.143289 18155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:18.143615 18155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:18.146812 18155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-1/data/instance:
uuid: "ddb33c23213d498cbe9a8130ef25827f"
format_stamp: "Formatted at 2026-08-10 23:25:18 on dist-test-slave-rmhg"
server_key: "60a7df2acc471069576e14e097706020"
server_key_iv: "1436fb535682ad20821dfbfb1e7f5b31"
server_key_version: "encryptionkey@0"
I20260810 23:25:18.147608 18155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance:
uuid: "ddb33c23213d498cbe9a8130ef25827f"
format_stamp: "Formatted at 2026-08-10 23:25:18 on dist-test-slave-rmhg"
server_key: "60a7df2acc471069576e14e097706020"
server_key_iv: "1436fb535682ad20821dfbfb1e7f5b31"
server_key_version: "encryptionkey@0"
I20260810 23:25:18.153363 18155 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 23:25:18.157677 18173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:18.159345 18155 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:25:18.159615 18155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-1/wal
uuid: "ddb33c23213d498cbe9a8130ef25827f"
format_stamp: "Formatted at 2026-08-10 23:25:18 on dist-test-slave-rmhg"
server_key: "60a7df2acc471069576e14e097706020"
server_key_iv: "1436fb535682ad20821dfbfb1e7f5b31"
server_key_version: "encryptionkey@0"
I20260810 23:25:18.159873 18155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:18.175203 18155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:18.180218 18155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:18.180547 18155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:18.183611 18155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:25:18.183743 18155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:18.183871 18155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:25:18.183951 18155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:25:18 dist-test-slave-rmhg krb5kdc[17740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404318, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:18.208511 17969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:18.184349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:52083 (local address 127.14.175.126:32813)
0810 23:25:18.185587 (+ 1238us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:18.185594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:18.186242 (+ 648us) server_negotiation.cc:407] Connection header received
0810 23:25:18.188743 (+ 2501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:18.188751 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:18.188855 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:18.188935 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:18.191316 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:18.192124 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:18.194780 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:18.195533 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:18.195698 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:18.201181 (+ 5483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:18.201189 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:18.201194 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:18.201248 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:18.204735 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:18.205825 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:18.205832 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:18.205834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:18.206001 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:18.206528 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:18.206535 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:18.206539 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:18.206867 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:18.207034 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:18.207901 (+ 867us) server_negotiation.cc:299] Negotiation successful
0810 23:25:18.208151 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1050}
I20260810 23:25:18.209914 18187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:18.185002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:32813 (local address 127.14.175.66:52083)
0810 23:25:18.186002 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:18.186069 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:18.188304 (+ 2235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:18.189561 (+ 1257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:18.189573 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:18.190438 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:18.191109 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:18.191131 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:18.192300 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:18.192307 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:18.194503 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:18.194522 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:18.195825 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:18.195835 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:18.196152 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:18.196939 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:18.196981 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:18.200809 (+ 3828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:18.204974 (+ 4165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:18.204987 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:18.205011 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:18.205658 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:18.206130 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:18.206135 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:18.206139 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:18.206395 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:18.207014 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:18.207024 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:18.207470 (+ 446us) client_negotiation.cc:769] Sending connection context
0810 23:25:18.207737 (+ 267us) client_negotiation.cc:240] Negotiation successful
0810 23:25:18.208350 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":476,"threads_started":1}
I20260810 23:25:18.233843 18278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:18.217391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:32813 (local address 127.14.175.66:42571)
0810 23:25:18.218180 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:18.218198 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:18.218444 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:18.218890 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:18.218895 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:18.219651 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:18.220109 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:18.220125 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:18.221608 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:18.221655 (+ 47us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:18.223204 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:18.223230 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:18.224488 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:18.224497 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:18.224710 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:18.225532 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:18.225553 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:18.226819 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:18.230472 (+ 3653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:18.230479 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:18.230484 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:18.230999 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:18.231527 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:18.231534 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:18.231538 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:18.231750 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:18.232531 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:18.232536 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:18.232757 (+ 221us) client_negotiation.cc:769] Sending connection context
0810 23:25:18.232919 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 23:25:18.233356 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":502,"threads_started":1}
I20260810 23:25:18.234010 17969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:18.217352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:42571 (local address 127.14.175.126:32813)
0810 23:25:18.217556 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:18.217561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:18.218317 (+ 756us) server_negotiation.cc:407] Connection header received
0810 23:25:18.218595 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:18.218601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:18.218704 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:18.218812 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:18.220318 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:18.221384 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:18.223463 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:18.224041 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:18.224183 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:18.227053 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:18.227058 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:18.227061 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:18.227110 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:18.230308 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:18.231125 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:18.231140 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:18.231145 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:18.231320 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:18.231913 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:18.231918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:18.231922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:18.232365 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:18.233323 (+ 958us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:18.233469 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 23:25:18.233703 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 23:25:18.242151 18155 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.66:45839
I20260810 23:25:18.242221 18347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.66:45839 every 8 connection(s)
I20260810 23:25:18.243522 18155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
I20260810 23:25:18.245728 18349 heartbeater.cc:347] Connected to a master server at 127.14.175.126:32813
I20260810 23:25:18.245990 18349 heartbeater.cc:464] Registering TS with master...
I20260810 23:25:18.246729 18349 heartbeater.cc:511] Master 127.14.175.126:32813 requested a full tablet report, sending...
I20260810 23:25:18.248246 17819 ts_manager.cc:194] Registered new tserver with Master: ddb33c23213d498cbe9a8130ef25827f (127.14.175.66:45839)
I20260810 23:25:18.249187 17819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.66@KRBTEST.COM'} at 127.14.175.66:52083
I20260810 23:25:18.251786 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 18155
I20260810 23:25:18.251983 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance
I20260810 23:25:18.252391 15037 external_mini_cluster.cc:1428] Setting key 4a8df500e66d3a437d443ecabd5a4a0a
WARNING: no policy specified for kudu/127.14.175.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
I20260810 23:25:18.301568 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.67:0
--local_ip_for_outbound_sockets=127.14.175.67
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:32813
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:32773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:18.648398 18354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:18.648747 18354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:18.648893 18354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:18.656517 18354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:18.656674 18354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:18.656811 18354 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 23:25:18.656883 18354 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:25:18.656989 18354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.67
I20260810 23:25:18.667435 18354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:32773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.175.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.175.126:32813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.67
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:18.668874 18354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:18.670336 18354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:18.684162 18361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:18.684191 18363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:18.684162 18360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:18.685120 18354 server_base.cc:1034] running on GCE node
I20260810 23:25:18.685917 18354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:18.687032 18354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:18.688292 18354 hybrid_clock.cc:648] HybridClock initialized: now 1786404318688227 us; error 55 us; skew 500 ppm
I20260810 23:25:18.688656 18354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:18 dist-test-slave-rmhg krb5kdc[17740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404318, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:18.692248 18354 init.cc:377] Logged in from keytab as kudu/127.14.175.67@KRBTEST.COM (short username kudu)
I20260810 23:25:18.694103 18354 webserver.cc:460] Webserver started at http://127.14.175.67:35911/ using document root <none> and password file <none>
I20260810 23:25:18.694841 18354 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:18.694970 18354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:18.695297 18354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:18.698616 18354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-2/data/instance:
uuid: "6b4d7412864047628e61a67a770e9a05"
format_stamp: "Formatted at 2026-08-10 23:25:18 on dist-test-slave-rmhg"
server_key: "3dda6f25e6e5ef16160f5bb75d0ce6ae"
server_key_iv: "60ccfcec88f288e017fef2ceb60b4198"
server_key_version: "encryptionkey@0"
I20260810 23:25:18.699496 18354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance:
uuid: "6b4d7412864047628e61a67a770e9a05"
format_stamp: "Formatted at 2026-08-10 23:25:18 on dist-test-slave-rmhg"
server_key: "3dda6f25e6e5ef16160f5bb75d0ce6ae"
server_key_iv: "60ccfcec88f288e017fef2ceb60b4198"
server_key_version: "encryptionkey@0"
I20260810 23:25:18.704869 18150 heartbeater.cc:503] Master 127.14.175.126:32813 was elected leader, sending a full tablet report...
I20260810 23:25:18.705613 18354 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260810 23:25:18.709767 18377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:18.711417 18354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:25:18.711591 18354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-2/wal
uuid: "6b4d7412864047628e61a67a770e9a05"
format_stamp: "Formatted at 2026-08-10 23:25:18 on dist-test-slave-rmhg"
server_key: "3dda6f25e6e5ef16160f5bb75d0ce6ae"
server_key_iv: "60ccfcec88f288e017fef2ceb60b4198"
server_key_version: "encryptionkey@0"
I20260810 23:25:18.711808 18354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:18.731465 18354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:18.736788 18354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:18.737100 18354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:18.740305 18354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:25:18.740422 18354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:18.740545 18354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:25:18.740617 18354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:25:18 dist-test-slave-rmhg krb5kdc[17740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404318, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:18.767211 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:18.741097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:59711 (local address 127.14.175.126:32813)
0810 23:25:18.741665 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:18.741672 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:18.743931 (+ 2259us) server_negotiation.cc:407] Connection header received
0810 23:25:18.746334 (+ 2403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:18.746340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:18.746421 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:18.746541 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:18.748915 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:18.750110 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:18.753207 (+ 3097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:18.753772 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:18.753903 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:18.758988 (+ 5085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:18.758996 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:18.759001 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:18.759061 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:18.762750 (+ 3689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:18.763979 (+ 1229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:18.763985 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:18.763987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:18.764177 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:18.764803 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:18.764808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:18.764811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:18.765139 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:18.765320 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:18.766490 (+ 1170us) server_negotiation.cc:299] Negotiation successful
0810 23:25:18.766771 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":210,"threads_started":1}
I20260810 23:25:18.768615 18397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:18.741850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:32813 (local address 127.14.175.67:59711)
0810 23:25:18.742955 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:18.743027 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:18.745896 (+ 2869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:18.746854 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:18.746864 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:18.747756 (+ 892us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:18.748691 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:18.748716 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:18.750311 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:18.750320 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:18.752995 (+ 2675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:18.753011 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:18.754050 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:18.754059 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:18.754417 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:18.755270 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:18.755296 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:18.758622 (+ 3326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:18.762964 (+ 4342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:18.762981 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:18.763002 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:18.763823 (+ 821us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:18.764337 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:18.764342 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:18.764346 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:18.764646 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:18.765410 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:18.765417 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:18.765986 (+ 569us) client_negotiation.cc:769] Sending connection context
0810 23:25:18.766352 (+ 366us) client_negotiation.cc:240] Negotiation successful
0810 23:25:18.766993 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":369,"threads_started":1}
I20260810 23:25:18.789431 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:18.775270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:32813 (local address 127.14.175.67:41769)
0810 23:25:18.775881 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:18.775902 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:18.776064 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:18.776421 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:18.776425 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:18.776945 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:18.777312 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:18.777325 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:18.778391 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:18.778397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:18.779475 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:18.779491 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:18.780279 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:18.780286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:18.780462 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:18.781242 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:18.781260 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:18.782599 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:18.786251 (+ 3652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:18.786255 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:18.786258 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:18.786908 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:18.787364 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:18.787368 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:18.787370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:18.787554 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:18.788369 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:18.788373 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:18.788580 (+ 207us) client_negotiation.cc:769] Sending connection context
0810 23:25:18.788719 (+ 139us) client_negotiation.cc:240] Negotiation successful
0810 23:25:18.789018 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":343,"threads_started":1}
I20260810 23:25:18.789592 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:18.775184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:41769 (local address 127.14.175.126:32813)
0810 23:25:18.775364 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:18.775368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:18.776008 (+ 640us) server_negotiation.cc:407] Connection header received
0810 23:25:18.776213 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:18.776217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:18.776289 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:18.776377 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:18.777472 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:18.778232 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:18.779641 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:18.780072 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:18.780192 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:18.782828 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:18.782834 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:18.782837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:18.782874 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:18.786017 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:18.787057 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:18.787064 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:18.787068 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:18.787231 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:18.787731 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:18.787738 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:18.787742 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:18.788196 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:18.788416 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:18.788846 (+ 430us) server_negotiation.cc:299] Negotiation successful
0810 23:25:18.789183 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:25:18.798274 18354 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.67:43999
I20260810 23:25:18.798322 18584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.67:43999 every 8 connection(s)
I20260810 23:25:18.799623 18354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
I20260810 23:25:18.801960 18586 heartbeater.cc:347] Connected to a master server at 127.14.175.126:32813
I20260810 23:25:18.802212 18586 heartbeater.cc:464] Registering TS with master...
I20260810 23:25:18.802948 18586 heartbeater.cc:511] Master 127.14.175.126:32813 requested a full tablet report, sending...
I20260810 23:25:18.804281 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 18354
I20260810 23:25:18.804502 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance
I20260810 23:25:18.804529 17819 ts_manager.cc:194] Registered new tserver with Master: 6b4d7412864047628e61a67a770e9a05 (127.14.175.67:43999)
I20260810 23:25:18.805001 15037 external_mini_cluster.cc:1428] Setting key 17f0450fcccfc53c3c25719d7726cc84
I20260810 23:25:18.805469 17819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.67@KRBTEST.COM'} at 127.14.175.67:59711
I20260810 23:25:18.808243 15037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 23:25:18 dist-test-slave-rmhg krb5kdc[17740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 23:25:18 dist-test-slave-rmhg krb5kdc[17740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:18.861304 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:18.847081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47402 (local address 127.14.175.126:32813)
0810 23:25:18.847265 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:18.847269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:18.847583 (+ 314us) server_negotiation.cc:407] Connection header received
0810 23:25:18.847772 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:18.847776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:18.847845 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:18.847942 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:18.849437 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:18.850233 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:18.851615 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:18.852053 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:18.852158 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:18.855992 (+ 3834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:18.855998 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:18.856001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:18.856040 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:18.858696 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:18.859354 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:18.859358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:18.859360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:18.859501 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:18.859975 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:18.859979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:18.859982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:18.860282 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:18.860491 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:18.860782 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 23:25:18.860972 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 23:25:18.865860 17819 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:25:18.872987 17819 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:25:18.913755 18619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:18.902412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:45839 (local address 127.0.0.1:52048)
0810 23:25:18.905785 (+ 3373us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:18.905806 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:18.905927 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:18.906509 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:18.906513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:18.906538 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:25:18.906869 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:18.906878 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:18.908210 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:18.908215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:18.910736 (+ 2521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:18.910747 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:18.912621 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:18.912628 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:18.912858 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:18.912921 (+ 63us) client_negotiation.cc:769] Sending connection context
0810 23:25:18.913062 (+ 141us) client_negotiation.cc:240] Negotiation successful
0810 23:25:18.913330 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3254,"mutex_wait_us":41,"thread_start_us":3053,"threads_started":1}
I20260810 23:25:18.914196 18608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:18.886340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:44481 (local address 127.0.0.1:37362)
0810 23:25:18.903536 (+ 17196us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:18.903560 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:18.903693 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:18.905945 (+ 2252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:18.905949 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:18.905977 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:25:18.906375 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:18.906387 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:18.908190 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:18.908195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:18.910411 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:18.910425 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:18.913420 (+ 2995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:18.913426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:18.913589 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:18.913623 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 23:25:18.913714 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 23:25:18.913845 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17032,"mutex_wait_us":90,"thread_start_us":11034,"threads_started":1}
I20260810 23:25:18.914304 18614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:18.886593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37362 (local address 127.14.175.65:44481)
0810 23:25:18.904253 (+ 17660us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:18.904263 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:18.904297 (+ 34us) server_negotiation.cc:407] Connection header received
0810 23:25:18.904399 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:18.904406 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:18.904692 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:18.904892 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:25:18.906570 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:18.908003 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:18.910591 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:18.912887 (+ 2296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:18.913026 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:18.913263 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:18.913822 (+ 559us) server_negotiation.cc:299] Negotiation successful
0810 23:25:18.914020 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17470,"thread_start_us":17322,"threads_started":1}
I20260810 23:25:18.914494 18609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:18.886170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:43999 (local address 127.0.0.1:40504)
0810 23:25:18.897812 (+ 11642us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:18.897863 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:18.898089 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:18.905092 (+ 7003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:18.905099 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:18.905129 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:25:18.905676 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:18.905692 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:18.907910 (+ 2218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:18.907916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:18.910677 (+ 2761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:18.910695 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:18.913009 (+ 2314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:18.913017 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:18.913264 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:18.913298 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 23:25:18.913373 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 23:25:18.913520 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11345,"mutex_wait_us":25,"thread_start_us":18411,"threads_started":1}
I20260810 23:25:18.914976 18616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:18.886561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40504 (local address 127.14.175.67:43999)
0810 23:25:18.902306 (+ 15745us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:18.902318 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:18.902365 (+ 47us) server_negotiation.cc:407] Connection header received
0810 23:25:18.902500 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:18.902509 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:18.902822 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:18.904891 (+ 2069us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:25:18.905975 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:18.907760 (+ 1785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:18.910886 (+ 3126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:18.912451 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:18.912580 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:18.914125 (+ 1545us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:18.914245 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 23:25:18.914699 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15523,"thread_start_us":15325,"threads_started":1}
I20260810 23:25:18.915809 18618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:18.898708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52048 (local address 127.14.175.66:45839)
0810 23:25:18.904658 (+ 5950us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:18.904667 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:18.905950 (+ 1283us) server_negotiation.cc:407] Connection header received
0810 23:25:18.906064 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:18.906071 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:18.906302 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:18.906502 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:25:18.907028 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:18.908022 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:18.910934 (+ 2912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:18.912099 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:18.912256 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:18.912462 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:18.914278 (+ 1816us) server_negotiation.cc:299] Negotiation successful
0810 23:25:18.915526 (+ 1248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5767,"thread_start_us":5595,"threads_started":1}
I20260810 23:25:18.918273 18469 tablet_service.cc:1467] Processing CreateTablet for tablet 4273016e26734c75852a2de73455ced2 (DEFAULT_TABLE table=test-table [id=3e2de5beca2c415faad4e4995f872776]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:25:18.918447 18042 tablet_service.cc:1467] Processing CreateTablet for tablet 4273016e26734c75852a2de73455ced2 (DEFAULT_TABLE table=test-table [id=3e2de5beca2c415faad4e4995f872776]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:25:18.919215 18250 tablet_service.cc:1467] Processing CreateTablet for tablet 4273016e26734c75852a2de73455ced2 (DEFAULT_TABLE table=test-table [id=3e2de5beca2c415faad4e4995f872776]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:25:18.919715 18469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4273016e26734c75852a2de73455ced2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:18.920320 18042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4273016e26734c75852a2de73455ced2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:18.920500 18250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4273016e26734c75852a2de73455ced2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:18.932085 18621 tablet_bootstrap.cc:492] T 4273016e26734c75852a2de73455ced2 P 6b4d7412864047628e61a67a770e9a05: Bootstrap starting.
I20260810 23:25:18.935563 18621 tablet_bootstrap.cc:654] T 4273016e26734c75852a2de73455ced2 P 6b4d7412864047628e61a67a770e9a05: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:18.936641 18621 log.cc:824] T 4273016e26734c75852a2de73455ced2 P 6b4d7412864047628e61a67a770e9a05: Log is configured to *not* fsync() on all Append() calls
I20260810 23:25:18.941648 18622 tablet_bootstrap.cc:492] T 4273016e26734c75852a2de73455ced2 P ddb33c23213d498cbe9a8130ef25827f: Bootstrap starting.
I20260810 23:25:18.942189 18623 tablet_bootstrap.cc:492] T 4273016e26734c75852a2de73455ced2 P deadca232e58457395f6519bdc10494e: Bootstrap starting.
I20260810 23:25:18.946372 18622 tablet_bootstrap.cc:654] T 4273016e26734c75852a2de73455ced2 P ddb33c23213d498cbe9a8130ef25827f: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:18.946993 18621 tablet_bootstrap.cc:492] T 4273016e26734c75852a2de73455ced2 P 6b4d7412864047628e61a67a770e9a05: No bootstrap required, opened a new log
I20260810 23:25:18.947194 18623 tablet_bootstrap.cc:654] T 4273016e26734c75852a2de73455ced2 P deadca232e58457395f6519bdc10494e: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:18.947322 18621 ts_tablet_manager.cc:1397] T 4273016e26734c75852a2de73455ced2 P 6b4d7412864047628e61a67a770e9a05: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20260810 23:25:18.947561 18622 log.cc:824] T 4273016e26734c75852a2de73455ced2 P ddb33c23213d498cbe9a8130ef25827f: Log is configured to *not* fsync() on all Append() calls
I20260810 23:25:18.948869 18623 log.cc:824] T 4273016e26734c75852a2de73455ced2 P deadca232e58457395f6519bdc10494e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:25:18.952491 18623 tablet_bootstrap.cc:492] T 4273016e26734c75852a2de73455ced2 P deadca232e58457395f6519bdc10494e: No bootstrap required, opened a new log
I20260810 23:25:18.952491 18622 tablet_bootstrap.cc:492] T 4273016e26734c75852a2de73455ced2 P ddb33c23213d498cbe9a8130ef25827f: No bootstrap required, opened a new log
I20260810 23:25:18.952827 18623 ts_tablet_manager.cc:1397] T 4273016e26734c75852a2de73455ced2 P deadca232e58457395f6519bdc10494e: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260810 23:25:18.952827 18622 ts_tablet_manager.cc:1397] T 4273016e26734c75852a2de73455ced2 P ddb33c23213d498cbe9a8130ef25827f: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20260810 23:25:18.955631 18621 raft_consensus.cc:348] T 4273016e26734c75852a2de73455ced2 P 6b4d7412864047628e61a67a770e9a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } } peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } }
I20260810 23:25:18.956043 18621 raft_consensus.cc:374] T 4273016e26734c75852a2de73455ced2 P 6b4d7412864047628e61a67a770e9a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:18.956141 18621 raft_consensus.cc:729] T 4273016e26734c75852a2de73455ced2 P 6b4d7412864047628e61a67a770e9a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b4d7412864047628e61a67a770e9a05, State: Initialized, Role: FOLLOWER
I20260810 23:25:18.956797 18621 consensus_queue.cc:260] T 4273016e26734c75852a2de73455ced2 P 6b4d7412864047628e61a67a770e9a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } } peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } }
I20260810 23:25:18.958036 18586 heartbeater.cc:503] Master 127.14.175.126:32813 was elected leader, sending a full tablet report...
I20260810 23:25:18.958796 18621 ts_tablet_manager.cc:1428] T 4273016e26734c75852a2de73455ced2 P 6b4d7412864047628e61a67a770e9a05: Time spent starting tablet: real 0.011s user 0.009s sys 0.003s
I20260810 23:25:18.963490 18623 raft_consensus.cc:348] T 4273016e26734c75852a2de73455ced2 P deadca232e58457395f6519bdc10494e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } } peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } }
I20260810 23:25:18.964040 18623 raft_consensus.cc:374] T 4273016e26734c75852a2de73455ced2 P deadca232e58457395f6519bdc10494e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:18.963712 18622 raft_consensus.cc:348] T 4273016e26734c75852a2de73455ced2 P ddb33c23213d498cbe9a8130ef25827f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } } peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } }
I20260810 23:25:18.964207 18622 raft_consensus.cc:374] T 4273016e26734c75852a2de73455ced2 P ddb33c23213d498cbe9a8130ef25827f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:18.964213 18623 raft_consensus.cc:729] T 4273016e26734c75852a2de73455ced2 P deadca232e58457395f6519bdc10494e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deadca232e58457395f6519bdc10494e, State: Initialized, Role: FOLLOWER
I20260810 23:25:18.964313 18622 raft_consensus.cc:729] T 4273016e26734c75852a2de73455ced2 P ddb33c23213d498cbe9a8130ef25827f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddb33c23213d498cbe9a8130ef25827f, State: Initialized, Role: FOLLOWER
I20260810 23:25:18.965008 18622 consensus_queue.cc:260] T 4273016e26734c75852a2de73455ced2 P ddb33c23213d498cbe9a8130ef25827f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } } peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } }
I20260810 23:25:18.965241 18623 consensus_queue.cc:260] T 4273016e26734c75852a2de73455ced2 P deadca232e58457395f6519bdc10494e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } } peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } }
I20260810 23:25:18.966384 18349 heartbeater.cc:503] Master 127.14.175.126:32813 was elected leader, sending a full tablet report...
I20260810 23:25:18.967156 18623 ts_tablet_manager.cc:1428] T 4273016e26734c75852a2de73455ced2 P deadca232e58457395f6519bdc10494e: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260810 23:25:18.967154 18622 ts_tablet_manager.cc:1428] T 4273016e26734c75852a2de73455ced2 P ddb33c23213d498cbe9a8130ef25827f: Time spent starting tablet: real 0.014s user 0.008s sys 0.007s
W20260810 23:25:18.995741 18350 tablet.cc:2369] T 4273016e26734c75852a2de73455ced2 P ddb33c23213d498cbe9a8130ef25827f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:25:19.051394 18587 tablet.cc:2369] T 4273016e26734c75852a2de73455ced2 P 6b4d7412864047628e61a67a770e9a05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:25:19.152570 18631 raft_consensus.cc:482] T 4273016e26734c75852a2de73455ced2 P ddb33c23213d498cbe9a8130ef25827f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:25:19.152911 18631 raft_consensus.cc:504] T 4273016e26734c75852a2de73455ced2 P ddb33c23213d498cbe9a8130ef25827f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } } peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } }
I20260810 23:25:19.154776 18631 leader_election.cc:288] T 4273016e26734c75852a2de73455ced2 P ddb33c23213d498cbe9a8130ef25827f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b4d7412864047628e61a67a770e9a05 (127.14.175.67:43999), deadca232e58457395f6519bdc10494e (127.14.175.65:44481)
I20260810 23:25:19.161762 18614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:19.155277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:58941 (local address 127.14.175.65:44481)
0810 23:25:19.155501 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:19.155509 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:19.156196 (+ 687us) server_negotiation.cc:407] Connection header received
0810 23:25:19.156338 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:19.156343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:19.156412 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:19.156497 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:25:19.157329 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.158119 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.159991 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.160769 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.160935 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:19.161014 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:19.161404 (+ 390us) server_negotiation.cc:299] Negotiation successful
0810 23:25:19.161533 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:25:19.161836 18616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:19.155371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:43099 (local address 127.14.175.67:43999)
0810 23:25:19.155561 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:19.155578 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:19.156054 (+ 476us) server_negotiation.cc:407] Connection header received
0810 23:25:19.156297 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:19.156302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:19.156385 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:19.156491 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:25:19.157413 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.158115 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.159988 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.160787 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.160944 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:19.161023 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:19.161447 (+ 424us) server_negotiation.cc:299] Negotiation successful
0810 23:25:19.161591 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 23:25:19.161947 18635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:19.155236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:44481 (local address 127.14.175.66:58941)
0810 23:25:19.156105 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:19.156120 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:19.156228 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:19.156522 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:19.156527 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:19.156602 (+ 75us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:25:19.157102 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:19.157118 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.158279 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.158284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:19.159813 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:19.159830 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.161043 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.161051 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:19.161249 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:19.161275 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 23:25:19.161361 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 23:25:19.161502 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"mutex_wait_us":55,"spinlock_wait_cycles":5632,"thread_start_us":528,"threads_started":1}
I20260810 23:25:19.162377 18633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:19.155073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:43999 (local address 127.14.175.66:43099)
0810 23:25:19.155915 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:19.155941 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:19.156131 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:19.156522 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:19.156527 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:19.156571 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:25:19.157263 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:19.157277 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.158255 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.158261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:19.159814 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:19.159830 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.161074 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.161081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:19.161293 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:19.161317 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 23:25:19.161399 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 23:25:19.161529 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":59,"spinlock_wait_cycles":2816,"thread_start_us":436,"threads_started":1}
I20260810 23:25:19.163017 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4273016e26734c75852a2de73455ced2" candidate_uuid: "ddb33c23213d498cbe9a8130ef25827f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deadca232e58457395f6519bdc10494e" is_pre_election: true
I20260810 23:25:19.163367 18507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4273016e26734c75852a2de73455ced2" candidate_uuid: "ddb33c23213d498cbe9a8130ef25827f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b4d7412864047628e61a67a770e9a05" is_pre_election: true
I20260810 23:25:19.163527 18079 raft_consensus.cc:2454] T 4273016e26734c75852a2de73455ced2 P deadca232e58457395f6519bdc10494e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddb33c23213d498cbe9a8130ef25827f in term 0.
I20260810 23:25:19.163944 18507 raft_consensus.cc:2454] T 4273016e26734c75852a2de73455ced2 P 6b4d7412864047628e61a67a770e9a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddb33c23213d498cbe9a8130ef25827f in term 0.
I20260810 23:25:19.164196 18179 leader_election.cc:302] T 4273016e26734c75852a2de73455ced2 P ddb33c23213d498cbe9a8130ef25827f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddb33c23213d498cbe9a8130ef25827f, deadca232e58457395f6519bdc10494e; no voters:
I20260810 23:25:19.164633 18631 raft_consensus.cc:2781] T 4273016e26734c75852a2de73455ced2 P ddb33c23213d498cbe9a8130ef25827f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:25:19.164784 18631 raft_consensus.cc:482] T 4273016e26734c75852a2de73455ced2 P ddb33c23213d498cbe9a8130ef25827f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:25:19.164885 18631 raft_consensus.cc:3037] T 4273016e26734c75852a2de73455ced2 P ddb33c23213d498cbe9a8130ef25827f [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:19.167309 18631 raft_consensus.cc:504] T 4273016e26734c75852a2de73455ced2 P ddb33c23213d498cbe9a8130ef25827f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } } peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } }
I20260810 23:25:19.167963 18631 leader_election.cc:288] T 4273016e26734c75852a2de73455ced2 P ddb33c23213d498cbe9a8130ef25827f [CANDIDATE]: Term 1 election: Requested vote from peers 6b4d7412864047628e61a67a770e9a05 (127.14.175.67:43999), deadca232e58457395f6519bdc10494e (127.14.175.65:44481)
I20260810 23:25:19.168505 18507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4273016e26734c75852a2de73455ced2" candidate_uuid: "ddb33c23213d498cbe9a8130ef25827f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b4d7412864047628e61a67a770e9a05"
I20260810 23:25:19.168702 18507 raft_consensus.cc:3037] T 4273016e26734c75852a2de73455ced2 P 6b4d7412864047628e61a67a770e9a05 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:19.170436 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4273016e26734c75852a2de73455ced2" candidate_uuid: "ddb33c23213d498cbe9a8130ef25827f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deadca232e58457395f6519bdc10494e"
I20260810 23:25:19.170670 18079 raft_consensus.cc:3037] T 4273016e26734c75852a2de73455ced2 P deadca232e58457395f6519bdc10494e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:19.171223 18507 raft_consensus.cc:2454] T 4273016e26734c75852a2de73455ced2 P 6b4d7412864047628e61a67a770e9a05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddb33c23213d498cbe9a8130ef25827f in term 1.
I20260810 23:25:19.171770 18174 leader_election.cc:302] T 4273016e26734c75852a2de73455ced2 P ddb33c23213d498cbe9a8130ef25827f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b4d7412864047628e61a67a770e9a05, ddb33c23213d498cbe9a8130ef25827f; no voters:
I20260810 23:25:19.172119 18631 raft_consensus.cc:2781] T 4273016e26734c75852a2de73455ced2 P ddb33c23213d498cbe9a8130ef25827f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:25:19.172623 18631 raft_consensus.cc:686] T 4273016e26734c75852a2de73455ced2 P ddb33c23213d498cbe9a8130ef25827f [term 1 LEADER]: Becoming Leader. State: Replica: ddb33c23213d498cbe9a8130ef25827f, State: Running, Role: LEADER
I20260810 23:25:19.173189 18631 consensus_queue.cc:237] T 4273016e26734c75852a2de73455ced2 P ddb33c23213d498cbe9a8130ef25827f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } } peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } }
I20260810 23:25:19.173561 18079 raft_consensus.cc:2454] T 4273016e26734c75852a2de73455ced2 P deadca232e58457395f6519bdc10494e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddb33c23213d498cbe9a8130ef25827f in term 1.
I20260810 23:25:19.178285 17817 catalog_manager.cc:5495] T 4273016e26734c75852a2de73455ced2 P ddb33c23213d498cbe9a8130ef25827f reported cstate change: term changed from 0 to 1, leader changed from <none> to ddb33c23213d498cbe9a8130ef25827f (127.14.175.66). New cstate: current_term: 1 leader_uuid: "ddb33c23213d498cbe9a8130ef25827f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } health_report { overall_health: HEALTHY } } }
W20260810 23:25:19.196020 18151 tablet.cc:2369] T 4273016e26734c75852a2de73455ced2 P deadca232e58457395f6519bdc10494e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 23:25:19 dist-test-slave-rmhg krb5kdc[17740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404316, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:19.250753 18650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:19.236077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:32813 (local address 127.0.0.1:47412)
0810 23:25:19.236668 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:19.236685 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:19.236817 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:19.237121 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:19.237124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:19.237874 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:19.238162 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:19.238174 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.239257 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.239261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:19.240254 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:19.240266 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.241055 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.241060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:19.241222 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:19.241930 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:19.241948 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:19.244350 (+ 2402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:19.248136 (+ 3786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:19.248149 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:19.248156 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:19.248622 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:19.248992 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:19.248996 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:19.248998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:19.249142 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:19.249775 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:19.249781 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:19.249966 (+ 185us) client_negotiation.cc:769] Sending connection context
0810 23:25:19.250123 (+ 157us) client_negotiation.cc:240] Negotiation successful
0810 23:25:19.250381 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":324,"threads_started":1}
I20260810 23:25:19.251157 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:19.236012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47412 (local address 127.14.175.126:32813)
0810 23:25:19.236208 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:19.236213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:19.236783 (+ 570us) server_negotiation.cc:407] Connection header received
0810 23:25:19.236920 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:19.236924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:19.237007 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:19.237094 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:19.238358 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.239129 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.240392 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.240883 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.240982 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:19.244581 (+ 3599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:19.244588 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:19.244593 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:19.244634 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:19.247970 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:19.248744 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:19.248750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:19.248754 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:19.248896 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:19.249260 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:19.249266 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:19.249270 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:19.249664 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:19.249801 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:19.250267 (+ 466us) server_negotiation.cc:299] Negotiation successful
0810 23:25:19.250553 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":54}
I20260810 23:25:19.256887 17817 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:47412:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 23:25:19.258038 17817 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:25:19.265185 18042 tablet_service.cc:1467] Processing CreateTablet for tablet 92980695d8d242ce80e860abfd6425d7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=be359bc350f14a2bbaf2f377f99ea4b0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:25:19.265511 18250 tablet_service.cc:1467] Processing CreateTablet for tablet 92980695d8d242ce80e860abfd6425d7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=be359bc350f14a2bbaf2f377f99ea4b0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:25:19.266458 18042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92980695d8d242ce80e860abfd6425d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:19.265892 18469 tablet_service.cc:1467] Processing CreateTablet for tablet 92980695d8d242ce80e860abfd6425d7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=be359bc350f14a2bbaf2f377f99ea4b0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:25:19.266485 18250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92980695d8d242ce80e860abfd6425d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:19.266801 18469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92980695d8d242ce80e860abfd6425d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:19.273021 18622 tablet_bootstrap.cc:492] T 92980695d8d242ce80e860abfd6425d7 P ddb33c23213d498cbe9a8130ef25827f: Bootstrap starting.
I20260810 23:25:19.274161 18621 tablet_bootstrap.cc:492] T 92980695d8d242ce80e860abfd6425d7 P 6b4d7412864047628e61a67a770e9a05: Bootstrap starting.
I20260810 23:25:19.274740 18622 tablet_bootstrap.cc:654] T 92980695d8d242ce80e860abfd6425d7 P ddb33c23213d498cbe9a8130ef25827f: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:19.274951 18623 tablet_bootstrap.cc:492] T 92980695d8d242ce80e860abfd6425d7 P deadca232e58457395f6519bdc10494e: Bootstrap starting.
I20260810 23:25:19.275960 18621 tablet_bootstrap.cc:654] T 92980695d8d242ce80e860abfd6425d7 P 6b4d7412864047628e61a67a770e9a05: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:19.276396 18623 tablet_bootstrap.cc:654] T 92980695d8d242ce80e860abfd6425d7 P deadca232e58457395f6519bdc10494e: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:19.278481 18621 tablet_bootstrap.cc:492] T 92980695d8d242ce80e860abfd6425d7 P 6b4d7412864047628e61a67a770e9a05: No bootstrap required, opened a new log
I20260810 23:25:19.278658 18621 ts_tablet_manager.cc:1397] T 92980695d8d242ce80e860abfd6425d7 P 6b4d7412864047628e61a67a770e9a05: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260810 23:25:19.278978 18623 tablet_bootstrap.cc:492] T 92980695d8d242ce80e860abfd6425d7 P deadca232e58457395f6519bdc10494e: No bootstrap required, opened a new log
I20260810 23:25:19.279162 18622 tablet_bootstrap.cc:492] T 92980695d8d242ce80e860abfd6425d7 P ddb33c23213d498cbe9a8130ef25827f: No bootstrap required, opened a new log
I20260810 23:25:19.279155 18623 ts_tablet_manager.cc:1397] T 92980695d8d242ce80e860abfd6425d7 P deadca232e58457395f6519bdc10494e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 23:25:19.279301 18622 ts_tablet_manager.cc:1397] T 92980695d8d242ce80e860abfd6425d7 P ddb33c23213d498cbe9a8130ef25827f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 23:25:19.279512 18621 raft_consensus.cc:348] T 92980695d8d242ce80e860abfd6425d7 P 6b4d7412864047628e61a67a770e9a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:19.279755 18621 raft_consensus.cc:374] T 92980695d8d242ce80e860abfd6425d7 P 6b4d7412864047628e61a67a770e9a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:19.279848 18621 raft_consensus.cc:729] T 92980695d8d242ce80e860abfd6425d7 P 6b4d7412864047628e61a67a770e9a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b4d7412864047628e61a67a770e9a05, State: Initialized, Role: FOLLOWER
I20260810 23:25:19.279933 18623 raft_consensus.cc:348] T 92980695d8d242ce80e860abfd6425d7 P deadca232e58457395f6519bdc10494e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:19.280068 18622 raft_consensus.cc:348] T 92980695d8d242ce80e860abfd6425d7 P ddb33c23213d498cbe9a8130ef25827f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:19.280083 18621 consensus_queue.cc:260] T 92980695d8d242ce80e860abfd6425d7 P 6b4d7412864047628e61a67a770e9a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:19.280221 18623 raft_consensus.cc:374] T 92980695d8d242ce80e860abfd6425d7 P deadca232e58457395f6519bdc10494e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:19.280315 18623 raft_consensus.cc:729] T 92980695d8d242ce80e860abfd6425d7 P deadca232e58457395f6519bdc10494e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deadca232e58457395f6519bdc10494e, State: Initialized, Role: FOLLOWER
I20260810 23:25:19.280555 18623 consensus_queue.cc:260] T 92980695d8d242ce80e860abfd6425d7 P deadca232e58457395f6519bdc10494e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:19.280740 18621 ts_tablet_manager.cc:1428] T 92980695d8d242ce80e860abfd6425d7 P 6b4d7412864047628e61a67a770e9a05: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 23:25:19.280671 18629 tablet_replica.cc:439] T 92980695d8d242ce80e860abfd6425d7 P 6b4d7412864047628e61a67a770e9a05: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } } }
I20260810 23:25:19.281028 18629 tablet_replica.cc:442] T 92980695d8d242ce80e860abfd6425d7 P 6b4d7412864047628e61a67a770e9a05: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:25:19.280301 18622 raft_consensus.cc:374] T 92980695d8d242ce80e860abfd6425d7 P ddb33c23213d498cbe9a8130ef25827f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:19.281703 18622 raft_consensus.cc:729] T 92980695d8d242ce80e860abfd6425d7 P ddb33c23213d498cbe9a8130ef25827f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddb33c23213d498cbe9a8130ef25827f, State: Initialized, Role: FOLLOWER
I20260810 23:25:19.281980 18622 consensus_queue.cc:260] T 92980695d8d242ce80e860abfd6425d7 P ddb33c23213d498cbe9a8130ef25827f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:19.282698 18622 ts_tablet_manager.cc:1428] T 92980695d8d242ce80e860abfd6425d7 P ddb33c23213d498cbe9a8130ef25827f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 23:25:19.283195 18623 ts_tablet_manager.cc:1428] T 92980695d8d242ce80e860abfd6425d7 P deadca232e58457395f6519bdc10494e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:25:19.283407 18632 tablet_replica.cc:439] T 92980695d8d242ce80e860abfd6425d7 P deadca232e58457395f6519bdc10494e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } } }
I20260810 23:25:19.283810 18632 tablet_replica.cc:442] T 92980695d8d242ce80e860abfd6425d7 P deadca232e58457395f6519bdc10494e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:25:19.282521 18631 tablet_replica.cc:439] T 92980695d8d242ce80e860abfd6425d7 P ddb33c23213d498cbe9a8130ef25827f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } } }
I20260810 23:25:19.284705 18631 tablet_replica.cc:442] T 92980695d8d242ce80e860abfd6425d7 P ddb33c23213d498cbe9a8130ef25827f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:25:19.522909 18631 raft_consensus.cc:482] T 92980695d8d242ce80e860abfd6425d7 P ddb33c23213d498cbe9a8130ef25827f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:25:19.523200 18631 raft_consensus.cc:504] T 92980695d8d242ce80e860abfd6425d7 P ddb33c23213d498cbe9a8130ef25827f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:19.523888 18631 leader_election.cc:288] T 92980695d8d242ce80e860abfd6425d7 P ddb33c23213d498cbe9a8130ef25827f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers deadca232e58457395f6519bdc10494e (127.14.175.65:44481), 6b4d7412864047628e61a67a770e9a05 (127.14.175.67:43999)
I20260810 23:25:19.524513 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92980695d8d242ce80e860abfd6425d7" candidate_uuid: "ddb33c23213d498cbe9a8130ef25827f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deadca232e58457395f6519bdc10494e" is_pre_election: true
I20260810 23:25:19.524518 18507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92980695d8d242ce80e860abfd6425d7" candidate_uuid: "ddb33c23213d498cbe9a8130ef25827f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b4d7412864047628e61a67a770e9a05" is_pre_election: true
I20260810 23:25:19.524766 18507 raft_consensus.cc:2454] T 92980695d8d242ce80e860abfd6425d7 P 6b4d7412864047628e61a67a770e9a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddb33c23213d498cbe9a8130ef25827f in term 0.
I20260810 23:25:19.524781 18079 raft_consensus.cc:2454] T 92980695d8d242ce80e860abfd6425d7 P deadca232e58457395f6519bdc10494e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddb33c23213d498cbe9a8130ef25827f in term 0.
I20260810 23:25:19.525290 18174 leader_election.cc:302] T 92980695d8d242ce80e860abfd6425d7 P ddb33c23213d498cbe9a8130ef25827f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6b4d7412864047628e61a67a770e9a05, ddb33c23213d498cbe9a8130ef25827f, deadca232e58457395f6519bdc10494e; no voters:
I20260810 23:25:19.525619 18631 raft_consensus.cc:2781] T 92980695d8d242ce80e860abfd6425d7 P ddb33c23213d498cbe9a8130ef25827f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:25:19.525720 18631 raft_consensus.cc:482] T 92980695d8d242ce80e860abfd6425d7 P ddb33c23213d498cbe9a8130ef25827f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:25:19.525832 18631 raft_consensus.cc:3037] T 92980695d8d242ce80e860abfd6425d7 P ddb33c23213d498cbe9a8130ef25827f [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:19.527930 18631 raft_consensus.cc:504] T 92980695d8d242ce80e860abfd6425d7 P ddb33c23213d498cbe9a8130ef25827f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:19.528666 18631 leader_election.cc:288] T 92980695d8d242ce80e860abfd6425d7 P ddb33c23213d498cbe9a8130ef25827f [CANDIDATE]: Term 1 election: Requested vote from peers deadca232e58457395f6519bdc10494e (127.14.175.65:44481), 6b4d7412864047628e61a67a770e9a05 (127.14.175.67:43999)
I20260810 23:25:19.529295 18079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92980695d8d242ce80e860abfd6425d7" candidate_uuid: "ddb33c23213d498cbe9a8130ef25827f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deadca232e58457395f6519bdc10494e"
I20260810 23:25:19.529309 18507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92980695d8d242ce80e860abfd6425d7" candidate_uuid: "ddb33c23213d498cbe9a8130ef25827f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b4d7412864047628e61a67a770e9a05"
I20260810 23:25:19.529551 18507 raft_consensus.cc:3037] T 92980695d8d242ce80e860abfd6425d7 P 6b4d7412864047628e61a67a770e9a05 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:19.529542 18079 raft_consensus.cc:3037] T 92980695d8d242ce80e860abfd6425d7 P deadca232e58457395f6519bdc10494e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:19.531584 18079 raft_consensus.cc:2454] T 92980695d8d242ce80e860abfd6425d7 P deadca232e58457395f6519bdc10494e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddb33c23213d498cbe9a8130ef25827f in term 1.
I20260810 23:25:19.531579 18507 raft_consensus.cc:2454] T 92980695d8d242ce80e860abfd6425d7 P 6b4d7412864047628e61a67a770e9a05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddb33c23213d498cbe9a8130ef25827f in term 1.
I20260810 23:25:19.532117 18179 leader_election.cc:302] T 92980695d8d242ce80e860abfd6425d7 P ddb33c23213d498cbe9a8130ef25827f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddb33c23213d498cbe9a8130ef25827f, deadca232e58457395f6519bdc10494e; no voters:
I20260810 23:25:19.532437 18631 raft_consensus.cc:2781] T 92980695d8d242ce80e860abfd6425d7 P ddb33c23213d498cbe9a8130ef25827f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:25:19.532617 18631 raft_consensus.cc:686] T 92980695d8d242ce80e860abfd6425d7 P ddb33c23213d498cbe9a8130ef25827f [term 1 LEADER]: Becoming Leader. State: Replica: ddb33c23213d498cbe9a8130ef25827f, State: Running, Role: LEADER
I20260810 23:25:19.532927 18631 consensus_queue.cc:237] T 92980695d8d242ce80e860abfd6425d7 P ddb33c23213d498cbe9a8130ef25827f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:19.534691 18637 tablet_replica.cc:439] T 92980695d8d242ce80e860abfd6425d7 P ddb33c23213d498cbe9a8130ef25827f: TxnStatusTablet state changed. Reason: New leader ddb33c23213d498cbe9a8130ef25827f. Latest consensus state: current_term: 1 leader_uuid: "ddb33c23213d498cbe9a8130ef25827f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } } }
I20260810 23:25:19.535087 18637 tablet_replica.cc:442] T 92980695d8d242ce80e860abfd6425d7 P ddb33c23213d498cbe9a8130ef25827f: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:25:19.535727 18657 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 23:25:19.536072 18657 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 23:25:19.536458 17814 catalog_manager.cc:5495] T 92980695d8d242ce80e860abfd6425d7 P ddb33c23213d498cbe9a8130ef25827f reported cstate change: term changed from 0 to 1, leader changed from <none> to ddb33c23213d498cbe9a8130ef25827f (127.14.175.66). New cstate: current_term: 1 leader_uuid: "ddb33c23213d498cbe9a8130ef25827f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } health_report { overall_health: UNKNOWN } } }
I20260810 23:25:19.582870 18650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:19.575515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:45839 (local address 127.0.0.1:52056)
0810 23:25:19.575834 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:19.575861 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:19.576067 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:19.576764 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:19.576767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:19.576778 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:25:19.576949 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:19.576956 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.578521 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.578526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:19.579572 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:19.579588 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.580722 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.580728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:19.580890 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:19.580979 (+ 89us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:25:19.582360 (+ 1381us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:25:19.582386 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 23:25:19.582471 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 23:25:19.582623 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260810 23:25:19.583050 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:19.575624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52056 (local address 127.14.175.66:45839)
0810 23:25:19.576280 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:19.576287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:19.576315 (+ 28us) server_negotiation.cc:407] Connection header received
0810 23:25:19.576420 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:19.576426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:19.576541 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:19.576735 (+ 194us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:25:19.577114 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.578372 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.579732 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.580421 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.580558 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:19.581273 (+ 715us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:25:19.582209 (+ 936us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:25:19.582378 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:19.582524 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 23:25:19.582624 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":309,"threads_started":1}
I20260810 23:25:19.591636 18507 raft_consensus.cc:1261] T 92980695d8d242ce80e860abfd6425d7 P 6b4d7412864047628e61a67a770e9a05 [term 1 FOLLOWER]: Refusing update from remote peer ddb33c23213d498cbe9a8130ef25827f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:25:19.591689 18079 raft_consensus.cc:1261] T 92980695d8d242ce80e860abfd6425d7 P deadca232e58457395f6519bdc10494e [term 1 FOLLOWER]: Refusing update from remote peer ddb33c23213d498cbe9a8130ef25827f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:25:19.592736 18631 consensus_queue.cc:1035] T 92980695d8d242ce80e860abfd6425d7 P ddb33c23213d498cbe9a8130ef25827f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:25:19.593214 18637 consensus_queue.cc:1035] T 92980695d8d242ce80e860abfd6425d7 P ddb33c23213d498cbe9a8130ef25827f [LEADER]: Connected to new peer: Peer: permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:25:19.601609 18629 tablet_replica.cc:439] T 92980695d8d242ce80e860abfd6425d7 P 6b4d7412864047628e61a67a770e9a05: TxnStatusTablet state changed. Reason: New leader ddb33c23213d498cbe9a8130ef25827f. Latest consensus state: current_term: 1 leader_uuid: "ddb33c23213d498cbe9a8130ef25827f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } } }
I20260810 23:25:19.601930 18629 tablet_replica.cc:442] T 92980695d8d242ce80e860abfd6425d7 P 6b4d7412864047628e61a67a770e9a05: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:25:19.602660 18637 consensus_queue.cc:1035] T 4273016e26734c75852a2de73455ced2 P ddb33c23213d498cbe9a8130ef25827f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:25:19.614431 18632 tablet_replica.cc:439] T 92980695d8d242ce80e860abfd6425d7 P deadca232e58457395f6519bdc10494e: TxnStatusTablet state changed. Reason: New leader ddb33c23213d498cbe9a8130ef25827f. Latest consensus state: current_term: 1 leader_uuid: "ddb33c23213d498cbe9a8130ef25827f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } } }
I20260810 23:25:19.614758 18632 tablet_replica.cc:442] T 92980695d8d242ce80e860abfd6425d7 P deadca232e58457395f6519bdc10494e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:25:19.620590 18631 tablet_replica.cc:439] T 92980695d8d242ce80e860abfd6425d7 P ddb33c23213d498cbe9a8130ef25827f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ddb33c23213d498cbe9a8130ef25827f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } } }
I20260810 23:25:19.628252 18631 tablet_replica.cc:442] T 92980695d8d242ce80e860abfd6425d7 P ddb33c23213d498cbe9a8130ef25827f: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:25:19.628372 18669 consensus_queue.cc:1035] T 4273016e26734c75852a2de73455ced2 P ddb33c23213d498cbe9a8130ef25827f [LEADER]: Connected to new peer: Peer: permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:25:19.629416 18637 tablet_replica.cc:439] T 92980695d8d242ce80e860abfd6425d7 P ddb33c23213d498cbe9a8130ef25827f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ddb33c23213d498cbe9a8130ef25827f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } } }
I20260810 23:25:19.629659 18637 tablet_replica.cc:442] T 92980695d8d242ce80e860abfd6425d7 P ddb33c23213d498cbe9a8130ef25827f: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:25:19.630815 18664 mvcc.cc:205] Tried to move back new op lower bound from 7317112093035302912 to 7317112092809428992. Current Snapshot: MvccSnapshot[applied={T|T < 7317112093035302912}]
I20260810 23:25:19.634464 18632 tablet_replica.cc:439] T 92980695d8d242ce80e860abfd6425d7 P deadca232e58457395f6519bdc10494e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ddb33c23213d498cbe9a8130ef25827f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } } }
I20260810 23:25:19.634771 18632 tablet_replica.cc:442] T 92980695d8d242ce80e860abfd6425d7 P deadca232e58457395f6519bdc10494e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:25:19.640276 18629 tablet_replica.cc:439] T 92980695d8d242ce80e860abfd6425d7 P 6b4d7412864047628e61a67a770e9a05: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ddb33c23213d498cbe9a8130ef25827f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } } }
I20260810 23:25:19.640591 18629 tablet_replica.cc:442] T 92980695d8d242ce80e860abfd6425d7 P 6b4d7412864047628e61a67a770e9a05: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:25:19.649749 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:19.640923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41342 (local address 127.14.175.66:45839)
0810 23:25:19.641352 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:19.641360 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:19.642671 (+ 1311us) server_negotiation.cc:407] Connection header received
0810 23:25:19.642992 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:19.642998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:19.643097 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:19.643165 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:25:19.643943 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.645266 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.647642 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.648321 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.648450 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:19.648954 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:25:19.649073 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:25:19.649181 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:19.649345 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 23:25:19.649434 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281}
I20260810 23:25:19.668969 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:19.662032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:33383 (local address 127.14.175.66:45839)
0810 23:25:19.662347 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:19.662353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:19.662949 (+ 596us) server_negotiation.cc:407] Connection header received
0810 23:25:19.663142 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:19.663148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:19.663257 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:19.663358 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:25:19.663806 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.665188 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.666749 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.667537 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.667668 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:19.668157 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:25:19.668310 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:25:19.668397 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:19.668528 (+ 131us) server_negotiation.cc:299] Negotiation successful
0810 23:25:19.668619 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 23:25:19.669075 18692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:19.662045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:45839 (local address 127.14.175.66:33383)
0810 23:25:19.662812 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:19.662840 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:19.663009 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:19.663390 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:19.663393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:19.663403 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:25:19.663647 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:19.663659 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.665332 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.665336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:19.666604 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:19.666619 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.667697 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.667702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:19.667830 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:19.667917 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:25:19.668418 (+ 501us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:25:19.668427 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 23:25:19.668486 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 23:25:19.668621 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"mutex_wait_us":2,"thread_start_us":480,"threads_started":1}
I20260810 23:25:19.747419 17814 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:25:19.747704 17814 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:25:19.751394 17814 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8420095ccbb64e43b12d94041d536850: Sending DeleteTablet for 3 replicas of tablet 4273016e26734c75852a2de73455ced2
I20260810 23:25:19.752776 18469 tablet_service.cc:1514] Processing DeleteTablet for tablet 4273016e26734c75852a2de73455ced2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:25:19 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:40504
I20260810 23:25:19.752776 18042 tablet_service.cc:1514] Processing DeleteTablet for tablet 4273016e26734c75852a2de73455ced2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:25:19 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:37362
I20260810 23:25:19.754169 18250 tablet_service.cc:1514] Processing DeleteTablet for tablet 4273016e26734c75852a2de73455ced2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:25:19 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:52048
I20260810 23:25:19.754577 17814 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:25:19.755306 17814 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:25:19.764180 18037 tablet_service.cc:1467] Processing CreateTablet for tablet 8cfc1f5c7dbe458e868c07d675e4a072 (DEFAULT_TABLE table=test-table [id=0eb1d3d5e7a24eecac704953918fa025]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:25:19.764863 18037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cfc1f5c7dbe458e868c07d675e4a072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:19.764848 18468 tablet_service.cc:1467] Processing CreateTablet for tablet 8cfc1f5c7dbe458e868c07d675e4a072 (DEFAULT_TABLE table=test-table [id=0eb1d3d5e7a24eecac704953918fa025]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:25:19.765497 18468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cfc1f5c7dbe458e868c07d675e4a072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:19.771132 18247 tablet_service.cc:1467] Processing CreateTablet for tablet 8cfc1f5c7dbe458e868c07d675e4a072 (DEFAULT_TABLE table=test-table [id=0eb1d3d5e7a24eecac704953918fa025]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:25:19.771827 18247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cfc1f5c7dbe458e868c07d675e4a072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:19.772432 18623 tablet_bootstrap.cc:492] T 8cfc1f5c7dbe458e868c07d675e4a072 P deadca232e58457395f6519bdc10494e: Bootstrap starting.
I20260810 23:25:19.774281 18623 tablet_bootstrap.cc:654] T 8cfc1f5c7dbe458e868c07d675e4a072 P deadca232e58457395f6519bdc10494e: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:19.781618 18622 tablet_bootstrap.cc:492] T 8cfc1f5c7dbe458e868c07d675e4a072 P ddb33c23213d498cbe9a8130ef25827f: Bootstrap starting.
I20260810 23:25:19.783789 18622 tablet_bootstrap.cc:654] T 8cfc1f5c7dbe458e868c07d675e4a072 P ddb33c23213d498cbe9a8130ef25827f: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:19.784997 18696 tablet_replica.cc:330] T 4273016e26734c75852a2de73455ced2 P ddb33c23213d498cbe9a8130ef25827f: stopping tablet replica
I20260810 23:25:19.785552 18696 raft_consensus.cc:2229] T 4273016e26734c75852a2de73455ced2 P ddb33c23213d498cbe9a8130ef25827f [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:25:19.786772 18696 raft_consensus.cc:2258] T 4273016e26734c75852a2de73455ced2 P ddb33c23213d498cbe9a8130ef25827f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:25:19.790149 18696 ts_tablet_manager.cc:1905] T 4273016e26734c75852a2de73455ced2 P ddb33c23213d498cbe9a8130ef25827f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:25:19.790915 18694 tablet_replica.cc:330] T 4273016e26734c75852a2de73455ced2 P deadca232e58457395f6519bdc10494e: stopping tablet replica
I20260810 23:25:19.791473 18694 raft_consensus.cc:2229] T 4273016e26734c75852a2de73455ced2 P deadca232e58457395f6519bdc10494e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:25:19.791874 18694 raft_consensus.cc:2258] T 4273016e26734c75852a2de73455ced2 P deadca232e58457395f6519bdc10494e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:25:19.792089 18623 tablet_bootstrap.cc:492] T 8cfc1f5c7dbe458e868c07d675e4a072 P deadca232e58457395f6519bdc10494e: No bootstrap required, opened a new log
I20260810 23:25:19.792752 18623 ts_tablet_manager.cc:1397] T 8cfc1f5c7dbe458e868c07d675e4a072 P deadca232e58457395f6519bdc10494e: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.011s
I20260810 23:25:19.793745 18623 raft_consensus.cc:348] T 8cfc1f5c7dbe458e868c07d675e4a072 P deadca232e58457395f6519bdc10494e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:19.794154 18623 raft_consensus.cc:374] T 8cfc1f5c7dbe458e868c07d675e4a072 P deadca232e58457395f6519bdc10494e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:19.794274 18623 raft_consensus.cc:729] T 8cfc1f5c7dbe458e868c07d675e4a072 P deadca232e58457395f6519bdc10494e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deadca232e58457395f6519bdc10494e, State: Initialized, Role: FOLLOWER
I20260810 23:25:19.794693 18623 consensus_queue.cc:260] T 8cfc1f5c7dbe458e868c07d675e4a072 P deadca232e58457395f6519bdc10494e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:19.799496 18694 ts_tablet_manager.cc:1905] T 4273016e26734c75852a2de73455ced2 P deadca232e58457395f6519bdc10494e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:25:19.800199 18622 tablet_bootstrap.cc:492] T 8cfc1f5c7dbe458e868c07d675e4a072 P ddb33c23213d498cbe9a8130ef25827f: No bootstrap required, opened a new log
I20260810 23:25:19.800491 18622 ts_tablet_manager.cc:1397] T 8cfc1f5c7dbe458e868c07d675e4a072 P ddb33c23213d498cbe9a8130ef25827f: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.004s
I20260810 23:25:19.801591 18622 raft_consensus.cc:348] T 8cfc1f5c7dbe458e868c07d675e4a072 P ddb33c23213d498cbe9a8130ef25827f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:19.804850 18700 tablet_bootstrap.cc:492] T 8cfc1f5c7dbe458e868c07d675e4a072 P 6b4d7412864047628e61a67a770e9a05: Bootstrap starting.
I20260810 23:25:19.805408 18622 raft_consensus.cc:374] T 8cfc1f5c7dbe458e868c07d675e4a072 P ddb33c23213d498cbe9a8130ef25827f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:19.805686 18622 raft_consensus.cc:729] T 8cfc1f5c7dbe458e868c07d675e4a072 P ddb33c23213d498cbe9a8130ef25827f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddb33c23213d498cbe9a8130ef25827f, State: Initialized, Role: FOLLOWER
I20260810 23:25:19.807894 18694 ts_tablet_manager.cc:1918] T 4273016e26734c75852a2de73455ced2 P deadca232e58457395f6519bdc10494e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:25:19.808085 18694 log.cc:1196] T 4273016e26734c75852a2de73455ced2 P deadca232e58457395f6519bdc10494e: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-0/wal/wals/4273016e26734c75852a2de73455ced2
I20260810 23:25:19.808262 18700 tablet_bootstrap.cc:654] T 8cfc1f5c7dbe458e868c07d675e4a072 P 6b4d7412864047628e61a67a770e9a05: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:19.808647 18694 ts_tablet_manager.cc:1939] T 4273016e26734c75852a2de73455ced2 P deadca232e58457395f6519bdc10494e: Deleting consensus metadata
I20260810 23:25:19.809657 18695 tablet_replica.cc:330] T 4273016e26734c75852a2de73455ced2 P 6b4d7412864047628e61a67a770e9a05: stopping tablet replica
I20260810 23:25:19.810119 18695 raft_consensus.cc:2229] T 4273016e26734c75852a2de73455ced2 P 6b4d7412864047628e61a67a770e9a05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:25:19.810442 18623 ts_tablet_manager.cc:1428] T 8cfc1f5c7dbe458e868c07d675e4a072 P deadca232e58457395f6519bdc10494e: Time spent starting tablet: real 0.017s user 0.000s sys 0.002s
I20260810 23:25:19.810506 18695 raft_consensus.cc:2258] T 4273016e26734c75852a2de73455ced2 P 6b4d7412864047628e61a67a770e9a05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:25:19.811062 17793 catalog_manager.cc:4841] TS deadca232e58457395f6519bdc10494e (127.14.175.65:44481): tablet 4273016e26734c75852a2de73455ced2 (table test-table [id=3e2de5beca2c415faad4e4995f872776]) successfully deleted
I20260810 23:25:19.806137 18622 consensus_queue.cc:260] T 8cfc1f5c7dbe458e868c07d675e4a072 P ddb33c23213d498cbe9a8130ef25827f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:19.812683 18695 ts_tablet_manager.cc:1905] T 4273016e26734c75852a2de73455ced2 P 6b4d7412864047628e61a67a770e9a05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:25:19.813405 18622 ts_tablet_manager.cc:1428] T 8cfc1f5c7dbe458e868c07d675e4a072 P ddb33c23213d498cbe9a8130ef25827f: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20260810 23:25:19.820751 18700 tablet_bootstrap.cc:492] T 8cfc1f5c7dbe458e868c07d675e4a072 P 6b4d7412864047628e61a67a770e9a05: No bootstrap required, opened a new log
I20260810 23:25:19.821050 18700 ts_tablet_manager.cc:1397] T 8cfc1f5c7dbe458e868c07d675e4a072 P 6b4d7412864047628e61a67a770e9a05: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20260810 23:25:19.822055 18695 ts_tablet_manager.cc:1918] T 4273016e26734c75852a2de73455ced2 P 6b4d7412864047628e61a67a770e9a05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:25:19.822194 18696 ts_tablet_manager.cc:1918] T 4273016e26734c75852a2de73455ced2 P ddb33c23213d498cbe9a8130ef25827f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:25:19.822127 18700 raft_consensus.cc:348] T 8cfc1f5c7dbe458e868c07d675e4a072 P 6b4d7412864047628e61a67a770e9a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:19.822378 18696 log.cc:1196] T 4273016e26734c75852a2de73455ced2 P ddb33c23213d498cbe9a8130ef25827f: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-1/wal/wals/4273016e26734c75852a2de73455ced2
I20260810 23:25:19.822382 18695 log.cc:1196] T 4273016e26734c75852a2de73455ced2 P 6b4d7412864047628e61a67a770e9a05: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-2/wal/wals/4273016e26734c75852a2de73455ced2
I20260810 23:25:19.822551 18700 raft_consensus.cc:374] T 8cfc1f5c7dbe458e868c07d675e4a072 P 6b4d7412864047628e61a67a770e9a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:19.822667 18700 raft_consensus.cc:729] T 8cfc1f5c7dbe458e868c07d675e4a072 P 6b4d7412864047628e61a67a770e9a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b4d7412864047628e61a67a770e9a05, State: Initialized, Role: FOLLOWER
I20260810 23:25:19.822957 18696 ts_tablet_manager.cc:1939] T 4273016e26734c75852a2de73455ced2 P ddb33c23213d498cbe9a8130ef25827f: Deleting consensus metadata
I20260810 23:25:19.823036 18700 consensus_queue.cc:260] T 8cfc1f5c7dbe458e868c07d675e4a072 P 6b4d7412864047628e61a67a770e9a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:19.823292 18695 ts_tablet_manager.cc:1939] T 4273016e26734c75852a2de73455ced2 P 6b4d7412864047628e61a67a770e9a05: Deleting consensus metadata
I20260810 23:25:19.825073 17793 catalog_manager.cc:4841] TS ddb33c23213d498cbe9a8130ef25827f (127.14.175.66:45839): tablet 4273016e26734c75852a2de73455ced2 (table test-table [id=3e2de5beca2c415faad4e4995f872776]) successfully deleted
I20260810 23:25:19.825609 18700 ts_tablet_manager.cc:1428] T 8cfc1f5c7dbe458e868c07d675e4a072 P 6b4d7412864047628e61a67a770e9a05: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260810 23:25:19.827373 17789 catalog_manager.cc:4841] TS 6b4d7412864047628e61a67a770e9a05 (127.14.175.67:43999): tablet 4273016e26734c75852a2de73455ced2 (table test-table [id=3e2de5beca2c415faad4e4995f872776]) successfully deleted
I20260810 23:25:19.835436 18629 raft_consensus.cc:482] T 8cfc1f5c7dbe458e868c07d675e4a072 P 6b4d7412864047628e61a67a770e9a05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:25:19.835702 18629 raft_consensus.cc:504] T 8cfc1f5c7dbe458e868c07d675e4a072 P 6b4d7412864047628e61a67a770e9a05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:19.837958 18629 leader_election.cc:288] T 8cfc1f5c7dbe458e868c07d675e4a072 P 6b4d7412864047628e61a67a770e9a05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers deadca232e58457395f6519bdc10494e (127.14.175.65:44481), ddb33c23213d498cbe9a8130ef25827f (127.14.175.66:45839)
I20260810 23:25:19.847599 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:19.838468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:44481 (local address 127.14.175.67:58717)
0810 23:25:19.839013 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:19.839034 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:19.839192 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:19.840815 (+ 1623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:19.840820 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:19.840858 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:25:19.841381 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:19.841396 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.842863 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.842869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:19.844755 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:19.844780 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.846614 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.846621 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:19.846823 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:19.846868 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 23:25:19.846996 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 23:25:19.847173 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":80,"thread_start_us":260,"threads_started":1}
I20260810 23:25:19.847618 18706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:19.838718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:58717 (local address 127.14.175.65:44481)
0810 23:25:19.840454 (+ 1736us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:19.840460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:19.840477 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:25:19.840545 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:19.840550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:19.840643 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:19.840751 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:25:19.841623 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.842714 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.844952 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.846352 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.846557 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:19.846643 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:19.847047 (+ 404us) server_negotiation.cc:299] Negotiation successful
0810 23:25:19.847223 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1590,"thread_start_us":1441,"threads_started":1}
I20260810 23:25:19.847643 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:19.839607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:34495 (local address 127.14.175.66:45839)
0810 23:25:19.839828 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:19.839833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:19.839857 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:25:19.840078 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:19.840084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:19.840181 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:19.840250 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:25:19.841195 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.842282 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.845069 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.846504 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.846668 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:19.846741 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:19.847233 (+ 492us) server_negotiation.cc:299] Negotiation successful
0810 23:25:19.847384 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 23:25:19.848445 18707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:19.839157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:45839 (local address 127.14.175.67:34495)
0810 23:25:19.839752 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:19.839772 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:19.839891 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:19.840487 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:19.840492 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:19.840519 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:25:19.840974 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:19.840991 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.842468 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.842474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:19.844867 (+ 2393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:19.844888 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.846871 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.846876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:19.847029 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:19.847063 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 23:25:19.847917 (+ 854us) client_negotiation.cc:240] Negotiation successful
0810 23:25:19.848056 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":77,"thread_start_us":704,"threads_started":1}
I20260810 23:25:19.849269 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cfc1f5c7dbe458e868c07d675e4a072" candidate_uuid: "6b4d7412864047628e61a67a770e9a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deadca232e58457395f6519bdc10494e" is_pre_election: true
I20260810 23:25:19.849407 18287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cfc1f5c7dbe458e868c07d675e4a072" candidate_uuid: "6b4d7412864047628e61a67a770e9a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb33c23213d498cbe9a8130ef25827f" is_pre_election: true
I20260810 23:25:19.849643 18075 raft_consensus.cc:2454] T 8cfc1f5c7dbe458e868c07d675e4a072 P deadca232e58457395f6519bdc10494e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b4d7412864047628e61a67a770e9a05 in term 0.
I20260810 23:25:19.849889 18287 raft_consensus.cc:2454] T 8cfc1f5c7dbe458e868c07d675e4a072 P ddb33c23213d498cbe9a8130ef25827f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b4d7412864047628e61a67a770e9a05 in term 0.
I20260810 23:25:19.850483 18388 leader_election.cc:302] T 8cfc1f5c7dbe458e868c07d675e4a072 P 6b4d7412864047628e61a67a770e9a05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b4d7412864047628e61a67a770e9a05, deadca232e58457395f6519bdc10494e; no voters:
I20260810 23:25:19.851001 18629 raft_consensus.cc:2781] T 8cfc1f5c7dbe458e868c07d675e4a072 P 6b4d7412864047628e61a67a770e9a05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:25:19.851110 18629 raft_consensus.cc:482] T 8cfc1f5c7dbe458e868c07d675e4a072 P 6b4d7412864047628e61a67a770e9a05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:25:19.851178 18629 raft_consensus.cc:3037] T 8cfc1f5c7dbe458e868c07d675e4a072 P 6b4d7412864047628e61a67a770e9a05 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:19.853698 18629 raft_consensus.cc:504] T 8cfc1f5c7dbe458e868c07d675e4a072 P 6b4d7412864047628e61a67a770e9a05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:19.854312 18629 leader_election.cc:288] T 8cfc1f5c7dbe458e868c07d675e4a072 P 6b4d7412864047628e61a67a770e9a05 [CANDIDATE]: Term 1 election: Requested vote from peers deadca232e58457395f6519bdc10494e (127.14.175.65:44481), ddb33c23213d498cbe9a8130ef25827f (127.14.175.66:45839)
I20260810 23:25:19.855003 18287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cfc1f5c7dbe458e868c07d675e4a072" candidate_uuid: "6b4d7412864047628e61a67a770e9a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb33c23213d498cbe9a8130ef25827f"
I20260810 23:25:19.855021 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cfc1f5c7dbe458e868c07d675e4a072" candidate_uuid: "6b4d7412864047628e61a67a770e9a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deadca232e58457395f6519bdc10494e"
I20260810 23:25:19.855237 18287 raft_consensus.cc:3037] T 8cfc1f5c7dbe458e868c07d675e4a072 P ddb33c23213d498cbe9a8130ef25827f [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:19.855276 18075 raft_consensus.cc:3037] T 8cfc1f5c7dbe458e868c07d675e4a072 P deadca232e58457395f6519bdc10494e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:19.857724 18287 raft_consensus.cc:2454] T 8cfc1f5c7dbe458e868c07d675e4a072 P ddb33c23213d498cbe9a8130ef25827f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b4d7412864047628e61a67a770e9a05 in term 1.
I20260810 23:25:19.858203 18388 leader_election.cc:302] T 8cfc1f5c7dbe458e868c07d675e4a072 P 6b4d7412864047628e61a67a770e9a05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b4d7412864047628e61a67a770e9a05, ddb33c23213d498cbe9a8130ef25827f; no voters:
I20260810 23:25:19.858410 18075 raft_consensus.cc:2454] T 8cfc1f5c7dbe458e868c07d675e4a072 P deadca232e58457395f6519bdc10494e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b4d7412864047628e61a67a770e9a05 in term 1.
I20260810 23:25:19.858506 18629 raft_consensus.cc:2781] T 8cfc1f5c7dbe458e868c07d675e4a072 P 6b4d7412864047628e61a67a770e9a05 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:25:19.859104 18629 raft_consensus.cc:686] T 8cfc1f5c7dbe458e868c07d675e4a072 P 6b4d7412864047628e61a67a770e9a05 [term 1 LEADER]: Becoming Leader. State: Replica: 6b4d7412864047628e61a67a770e9a05, State: Running, Role: LEADER
I20260810 23:25:19.859526 18629 consensus_queue.cc:237] T 8cfc1f5c7dbe458e868c07d675e4a072 P 6b4d7412864047628e61a67a770e9a05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:19.863418 17819 catalog_manager.cc:5495] T 8cfc1f5c7dbe458e868c07d675e4a072 P 6b4d7412864047628e61a67a770e9a05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b4d7412864047628e61a67a770e9a05 (127.14.175.67). New cstate: current_term: 1 leader_uuid: "6b4d7412864047628e61a67a770e9a05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } health_report { overall_health: HEALTHY } } }
I20260810 23:25:19.888906 18712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:19.882631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49950 (local address 127.14.175.67:43999)
0810 23:25:19.883486 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:19.883491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:19.883511 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:25:19.883603 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:19.883606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:19.883720 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:19.883881 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:25:19.884357 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.885228 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.886613 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.887035 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.887149 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:19.887625 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:25:19.888218 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:25:19.888366 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:19.888476 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 23:25:19.888565 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":599,"threads_started":1}
I20260810 23:25:19.906679 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:19.900435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:47953 (local address 127.14.175.66:45839)
0810 23:25:19.900593 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:19.900598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:19.901279 (+ 681us) server_negotiation.cc:407] Connection header received
0810 23:25:19.901502 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:19.901506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:19.901584 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:19.901685 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:25:19.902178 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.902991 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.904564 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.905198 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.905325 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:19.905868 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:25:19.906010 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:25:19.906108 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:19.906296 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 23:25:19.906380 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 23:25:19.906865 18719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:19.900478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:45839 (local address 127.14.175.67:47953)
0810 23:25:19.901136 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:19.901184 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:19.901355 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:19.901710 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:19.901714 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:19.901727 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:25:19.902028 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:19.902038 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.903146 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.903150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:19.904398 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:19.904413 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.905389 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.905396 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:19.905551 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:19.905621 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:25:19.906153 (+ 532us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:25:19.906168 (+ 15us) client_negotiation.cc:769] Sending connection context
0810 23:25:19.906274 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 23:25:19.906440 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":315,"threads_started":1}
I20260810 23:25:19.916812 18075 raft_consensus.cc:1261] T 8cfc1f5c7dbe458e868c07d675e4a072 P deadca232e58457395f6519bdc10494e [term 1 FOLLOWER]: Refusing update from remote peer 6b4d7412864047628e61a67a770e9a05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:25:19.917121 18287 raft_consensus.cc:1261] T 8cfc1f5c7dbe458e868c07d675e4a072 P ddb33c23213d498cbe9a8130ef25827f [term 1 FOLLOWER]: Refusing update from remote peer 6b4d7412864047628e61a67a770e9a05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:25:19.917559 18709 consensus_queue.cc:1035] T 8cfc1f5c7dbe458e868c07d675e4a072 P 6b4d7412864047628e61a67a770e9a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:25:19.918402 18709 consensus_queue.cc:1035] T 8cfc1f5c7dbe458e868c07d675e4a072 P 6b4d7412864047628e61a67a770e9a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:25:19.926332 18665 mvcc.cc:205] Tried to move back new op lower bound from 7317112094372093952 to 7317112094148362240. Current Snapshot: MvccSnapshot[applied={T|T < 7317112094372093952}]
I20260810 23:25:19.930521 18663 mvcc.cc:205] Tried to move back new op lower bound from 7317112094372093952 to 7317112094148362240. Current Snapshot: MvccSnapshot[applied={T|T < 7317112094372093952}]
I20260810 23:25:19.951383 18712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:19.944863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:40609 (local address 127.14.175.67:43999)
0810 23:25:19.945385 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:19.945391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:19.945407 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:25:19.945471 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:19.945476 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:19.945562 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:19.945675 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:25:19.946230 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.947290 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.949326 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.949913 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.950050 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:19.950578 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:25:19.950764 (+ 186us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:25:19.950861 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:19.951005 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 23:25:19.951096 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391}
I20260810 23:25:19.951411 18692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:19.944611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:43999 (local address 127.14.175.66:40609)
0810 23:25:19.944973 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:19.945001 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:19.945124 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:19.945859 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:19.945865 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:19.945879 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:25:19.946083 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:19.946093 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.947463 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.947470 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:19.949080 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:19.949098 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:19.950085 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:19.950092 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:19.950298 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:19.950357 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:25:19.950874 (+ 517us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:25:19.950887 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 23:25:19.950975 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 23:25:19.951133 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196}
W20260810 23:25:19.952740 18664 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 23:25:21.572021 17814 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:25:21.572326 17814 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:25:21.576903 17814 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8420095ccbb64e43b12d94041d536850: Sending DeleteTablet for 3 replicas of tablet 8cfc1f5c7dbe458e868c07d675e4a072
I20260810 23:25:21.578099 18247 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cfc1f5c7dbe458e868c07d675e4a072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:25:21 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:52048
I20260810 23:25:21.578127 18042 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cfc1f5c7dbe458e868c07d675e4a072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:25:21 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:37362
I20260810 23:25:21.578099 18469 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cfc1f5c7dbe458e868c07d675e4a072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:25:21 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:40504
I20260810 23:25:21.590579 18765 tablet_replica.cc:330] T 8cfc1f5c7dbe458e868c07d675e4a072 P ddb33c23213d498cbe9a8130ef25827f: stopping tablet replica
I20260810 23:25:21.591004 18765 raft_consensus.cc:2229] T 8cfc1f5c7dbe458e868c07d675e4a072 P ddb33c23213d498cbe9a8130ef25827f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:25:21.591403 18765 raft_consensus.cc:2258] T 8cfc1f5c7dbe458e868c07d675e4a072 P ddb33c23213d498cbe9a8130ef25827f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:25:21.592810 18765 ts_tablet_manager.cc:1905] T 8cfc1f5c7dbe458e868c07d675e4a072 P ddb33c23213d498cbe9a8130ef25827f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:25:21.592972 18760 tablet_replica.cc:330] T 8cfc1f5c7dbe458e868c07d675e4a072 P deadca232e58457395f6519bdc10494e: stopping tablet replica
I20260810 23:25:21.593362 18763 tablet_replica.cc:330] T 8cfc1f5c7dbe458e868c07d675e4a072 P 6b4d7412864047628e61a67a770e9a05: stopping tablet replica
I20260810 23:25:21.593381 18760 raft_consensus.cc:2229] T 8cfc1f5c7dbe458e868c07d675e4a072 P deadca232e58457395f6519bdc10494e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:25:21.593633 18763 raft_consensus.cc:2229] T 8cfc1f5c7dbe458e868c07d675e4a072 P 6b4d7412864047628e61a67a770e9a05 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:25:21.593659 18760 raft_consensus.cc:2258] T 8cfc1f5c7dbe458e868c07d675e4a072 P deadca232e58457395f6519bdc10494e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:25:21.594352 18763 raft_consensus.cc:2258] T 8cfc1f5c7dbe458e868c07d675e4a072 P 6b4d7412864047628e61a67a770e9a05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:25:21.594770 18760 ts_tablet_manager.cc:1905] T 8cfc1f5c7dbe458e868c07d675e4a072 P deadca232e58457395f6519bdc10494e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:25:21.595332 18763 ts_tablet_manager.cc:1905] T 8cfc1f5c7dbe458e868c07d675e4a072 P 6b4d7412864047628e61a67a770e9a05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:25:21.600735 18765 ts_tablet_manager.cc:1918] T 8cfc1f5c7dbe458e868c07d675e4a072 P ddb33c23213d498cbe9a8130ef25827f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 23:25:21.601085 18765 log.cc:1196] T 8cfc1f5c7dbe458e868c07d675e4a072 P ddb33c23213d498cbe9a8130ef25827f: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-1/wal/wals/8cfc1f5c7dbe458e868c07d675e4a072
I20260810 23:25:21.601931 18765 ts_tablet_manager.cc:1939] T 8cfc1f5c7dbe458e868c07d675e4a072 P ddb33c23213d498cbe9a8130ef25827f: Deleting consensus metadata
I20260810 23:25:21.603142 18760 ts_tablet_manager.cc:1918] T 8cfc1f5c7dbe458e868c07d675e4a072 P deadca232e58457395f6519bdc10494e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 23:25:21.603236 18763 ts_tablet_manager.cc:1918] T 8cfc1f5c7dbe458e868c07d675e4a072 P 6b4d7412864047628e61a67a770e9a05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 23:25:21.603443 18760 log.cc:1196] T 8cfc1f5c7dbe458e868c07d675e4a072 P deadca232e58457395f6519bdc10494e: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-0/wal/wals/8cfc1f5c7dbe458e868c07d675e4a072
I20260810 23:25:21.603415 17793 catalog_manager.cc:4841] TS ddb33c23213d498cbe9a8130ef25827f (127.14.175.66:45839): tablet 8cfc1f5c7dbe458e868c07d675e4a072 (table test-table [id=0eb1d3d5e7a24eecac704953918fa025]) successfully deleted
I20260810 23:25:21.603502 18763 log.cc:1196] T 8cfc1f5c7dbe458e868c07d675e4a072 P 6b4d7412864047628e61a67a770e9a05: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-2/wal/wals/8cfc1f5c7dbe458e868c07d675e4a072
I20260810 23:25:21.604259 18760 ts_tablet_manager.cc:1939] T 8cfc1f5c7dbe458e868c07d675e4a072 P deadca232e58457395f6519bdc10494e: Deleting consensus metadata
I20260810 23:25:21.604295 18763 ts_tablet_manager.cc:1939] T 8cfc1f5c7dbe458e868c07d675e4a072 P 6b4d7412864047628e61a67a770e9a05: Deleting consensus metadata
I20260810 23:25:21.605645 17789 catalog_manager.cc:4841] TS 6b4d7412864047628e61a67a770e9a05 (127.14.175.67:43999): tablet 8cfc1f5c7dbe458e868c07d675e4a072 (table test-table [id=0eb1d3d5e7a24eecac704953918fa025]) successfully deleted
I20260810 23:25:21.605645 17793 catalog_manager.cc:4841] TS deadca232e58457395f6519bdc10494e (127.14.175.65:44481): tablet 8cfc1f5c7dbe458e868c07d675e4a072 (table test-table [id=0eb1d3d5e7a24eecac704953918fa025]) successfully deleted
Aug 10 23:25:21 dist-test-slave-rmhg krb5kdc[17740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404321, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 23:25:21 dist-test-slave-rmhg krb5kdc[17740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404321, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:21.649539 18781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:21.629064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55698 (local address 127.14.175.126:32813)
0810 23:25:21.629846 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:21.629853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:21.630243 (+ 390us) server_negotiation.cc:407] Connection header received
0810 23:25:21.630435 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:21.630439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:21.630521 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:21.630622 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:21.632746 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:21.633963 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:21.635442 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:21.636157 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:21.636267 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:21.641667 (+ 5400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:21.641674 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:21.641677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:21.641722 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:21.646125 (+ 4403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:21.647000 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:21.647004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:21.647006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:21.647255 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:21.647744 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:21.647747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:21.647749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:21.648234 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:21.648487 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:21.648789 (+ 302us) server_negotiation.cc:299] Negotiation successful
0810 23:25:21.649001 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":402,"threads_started":1}
I20260810 23:25:21.653542 17814 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:25:21.654430 17814 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:25:21.662331 18247 tablet_service.cc:1467] Processing CreateTablet for tablet fc37ae55dce445ccbb451cb21d9f03f0 (DEFAULT_TABLE table=test-table [id=c4c40292fd9e43a89894eff13434032c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:25:21.662318 18042 tablet_service.cc:1467] Processing CreateTablet for tablet fc37ae55dce445ccbb451cb21d9f03f0 (DEFAULT_TABLE table=test-table [id=c4c40292fd9e43a89894eff13434032c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:25:21.662724 18469 tablet_service.cc:1467] Processing CreateTablet for tablet fc37ae55dce445ccbb451cb21d9f03f0 (DEFAULT_TABLE table=test-table [id=c4c40292fd9e43a89894eff13434032c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:25:21.662909 18247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc37ae55dce445ccbb451cb21d9f03f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:21.662904 18042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc37ae55dce445ccbb451cb21d9f03f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:21.663213 18469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc37ae55dce445ccbb451cb21d9f03f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:21.673030 18785 tablet_bootstrap.cc:492] T fc37ae55dce445ccbb451cb21d9f03f0 P deadca232e58457395f6519bdc10494e: Bootstrap starting.
I20260810 23:25:21.671954 18782 tablet_bootstrap.cc:492] T fc37ae55dce445ccbb451cb21d9f03f0 P 6b4d7412864047628e61a67a770e9a05: Bootstrap starting.
I20260810 23:25:21.673941 18787 tablet_bootstrap.cc:492] T fc37ae55dce445ccbb451cb21d9f03f0 P ddb33c23213d498cbe9a8130ef25827f: Bootstrap starting.
I20260810 23:25:21.675247 18785 tablet_bootstrap.cc:654] T fc37ae55dce445ccbb451cb21d9f03f0 P deadca232e58457395f6519bdc10494e: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:21.675774 18782 tablet_bootstrap.cc:654] T fc37ae55dce445ccbb451cb21d9f03f0 P 6b4d7412864047628e61a67a770e9a05: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:21.675943 18787 tablet_bootstrap.cc:654] T fc37ae55dce445ccbb451cb21d9f03f0 P ddb33c23213d498cbe9a8130ef25827f: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:21.678340 18787 tablet_bootstrap.cc:492] T fc37ae55dce445ccbb451cb21d9f03f0 P ddb33c23213d498cbe9a8130ef25827f: No bootstrap required, opened a new log
I20260810 23:25:21.678478 18787 ts_tablet_manager.cc:1397] T fc37ae55dce445ccbb451cb21d9f03f0 P ddb33c23213d498cbe9a8130ef25827f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 23:25:21.678646 18785 tablet_bootstrap.cc:492] T fc37ae55dce445ccbb451cb21d9f03f0 P deadca232e58457395f6519bdc10494e: No bootstrap required, opened a new log
I20260810 23:25:21.678843 18785 ts_tablet_manager.cc:1397] T fc37ae55dce445ccbb451cb21d9f03f0 P deadca232e58457395f6519bdc10494e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 23:25:21.679234 18787 raft_consensus.cc:348] T fc37ae55dce445ccbb451cb21d9f03f0 P ddb33c23213d498cbe9a8130ef25827f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:21.679412 18782 tablet_bootstrap.cc:492] T fc37ae55dce445ccbb451cb21d9f03f0 P 6b4d7412864047628e61a67a770e9a05: No bootstrap required, opened a new log
I20260810 23:25:21.679445 18787 raft_consensus.cc:374] T fc37ae55dce445ccbb451cb21d9f03f0 P ddb33c23213d498cbe9a8130ef25827f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:21.679571 18782 ts_tablet_manager.cc:1397] T fc37ae55dce445ccbb451cb21d9f03f0 P 6b4d7412864047628e61a67a770e9a05: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260810 23:25:21.679572 18787 raft_consensus.cc:729] T fc37ae55dce445ccbb451cb21d9f03f0 P ddb33c23213d498cbe9a8130ef25827f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddb33c23213d498cbe9a8130ef25827f, State: Initialized, Role: FOLLOWER
I20260810 23:25:21.679698 18785 raft_consensus.cc:348] T fc37ae55dce445ccbb451cb21d9f03f0 P deadca232e58457395f6519bdc10494e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:21.679823 18787 consensus_queue.cc:260] T fc37ae55dce445ccbb451cb21d9f03f0 P ddb33c23213d498cbe9a8130ef25827f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:21.679955 18785 raft_consensus.cc:374] T fc37ae55dce445ccbb451cb21d9f03f0 P deadca232e58457395f6519bdc10494e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:21.680052 18785 raft_consensus.cc:729] T fc37ae55dce445ccbb451cb21d9f03f0 P deadca232e58457395f6519bdc10494e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deadca232e58457395f6519bdc10494e, State: Initialized, Role: FOLLOWER
I20260810 23:25:21.680349 18785 consensus_queue.cc:260] T fc37ae55dce445ccbb451cb21d9f03f0 P deadca232e58457395f6519bdc10494e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:21.680482 18787 ts_tablet_manager.cc:1428] T fc37ae55dce445ccbb451cb21d9f03f0 P ddb33c23213d498cbe9a8130ef25827f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:25:21.680377 18782 raft_consensus.cc:348] T fc37ae55dce445ccbb451cb21d9f03f0 P 6b4d7412864047628e61a67a770e9a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:21.680640 18782 raft_consensus.cc:374] T fc37ae55dce445ccbb451cb21d9f03f0 P 6b4d7412864047628e61a67a770e9a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:21.680780 18782 raft_consensus.cc:729] T fc37ae55dce445ccbb451cb21d9f03f0 P 6b4d7412864047628e61a67a770e9a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b4d7412864047628e61a67a770e9a05, State: Initialized, Role: FOLLOWER
I20260810 23:25:21.681087 18782 consensus_queue.cc:260] T fc37ae55dce445ccbb451cb21d9f03f0 P 6b4d7412864047628e61a67a770e9a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:21.681938 18785 ts_tablet_manager.cc:1428] T fc37ae55dce445ccbb451cb21d9f03f0 P deadca232e58457395f6519bdc10494e: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260810 23:25:21.682760 18782 ts_tablet_manager.cc:1428] T fc37ae55dce445ccbb451cb21d9f03f0 P 6b4d7412864047628e61a67a770e9a05: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 23:25:21.721577 18722 raft_consensus.cc:482] T fc37ae55dce445ccbb451cb21d9f03f0 P 6b4d7412864047628e61a67a770e9a05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:25:21.721858 18722 raft_consensus.cc:504] T fc37ae55dce445ccbb451cb21d9f03f0 P 6b4d7412864047628e61a67a770e9a05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:21.722551 18722 leader_election.cc:288] T fc37ae55dce445ccbb451cb21d9f03f0 P 6b4d7412864047628e61a67a770e9a05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers deadca232e58457395f6519bdc10494e (127.14.175.65:44481), ddb33c23213d498cbe9a8130ef25827f (127.14.175.66:45839)
I20260810 23:25:21.723285 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc37ae55dce445ccbb451cb21d9f03f0" candidate_uuid: "6b4d7412864047628e61a67a770e9a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deadca232e58457395f6519bdc10494e" is_pre_election: true
I20260810 23:25:21.723341 18287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc37ae55dce445ccbb451cb21d9f03f0" candidate_uuid: "6b4d7412864047628e61a67a770e9a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb33c23213d498cbe9a8130ef25827f" is_pre_election: true
I20260810 23:25:21.723534 18075 raft_consensus.cc:2454] T fc37ae55dce445ccbb451cb21d9f03f0 P deadca232e58457395f6519bdc10494e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b4d7412864047628e61a67a770e9a05 in term 0.
I20260810 23:25:21.723577 18287 raft_consensus.cc:2454] T fc37ae55dce445ccbb451cb21d9f03f0 P ddb33c23213d498cbe9a8130ef25827f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b4d7412864047628e61a67a770e9a05 in term 0.
I20260810 23:25:21.724053 18388 leader_election.cc:302] T fc37ae55dce445ccbb451cb21d9f03f0 P 6b4d7412864047628e61a67a770e9a05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b4d7412864047628e61a67a770e9a05, deadca232e58457395f6519bdc10494e; no voters:
I20260810 23:25:21.724349 18722 raft_consensus.cc:2781] T fc37ae55dce445ccbb451cb21d9f03f0 P 6b4d7412864047628e61a67a770e9a05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:25:21.724483 18722 raft_consensus.cc:482] T fc37ae55dce445ccbb451cb21d9f03f0 P 6b4d7412864047628e61a67a770e9a05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:25:21.724571 18722 raft_consensus.cc:3037] T fc37ae55dce445ccbb451cb21d9f03f0 P 6b4d7412864047628e61a67a770e9a05 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:21.726879 18722 raft_consensus.cc:504] T fc37ae55dce445ccbb451cb21d9f03f0 P 6b4d7412864047628e61a67a770e9a05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:21.727522 18722 leader_election.cc:288] T fc37ae55dce445ccbb451cb21d9f03f0 P 6b4d7412864047628e61a67a770e9a05 [CANDIDATE]: Term 1 election: Requested vote from peers deadca232e58457395f6519bdc10494e (127.14.175.65:44481), ddb33c23213d498cbe9a8130ef25827f (127.14.175.66:45839)
I20260810 23:25:21.728250 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc37ae55dce445ccbb451cb21d9f03f0" candidate_uuid: "6b4d7412864047628e61a67a770e9a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deadca232e58457395f6519bdc10494e"
I20260810 23:25:21.728302 18287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc37ae55dce445ccbb451cb21d9f03f0" candidate_uuid: "6b4d7412864047628e61a67a770e9a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb33c23213d498cbe9a8130ef25827f"
I20260810 23:25:21.728485 18075 raft_consensus.cc:3037] T fc37ae55dce445ccbb451cb21d9f03f0 P deadca232e58457395f6519bdc10494e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:21.728523 18287 raft_consensus.cc:3037] T fc37ae55dce445ccbb451cb21d9f03f0 P ddb33c23213d498cbe9a8130ef25827f [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:21.730808 18075 raft_consensus.cc:2454] T fc37ae55dce445ccbb451cb21d9f03f0 P deadca232e58457395f6519bdc10494e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b4d7412864047628e61a67a770e9a05 in term 1.
I20260810 23:25:21.730793 18287 raft_consensus.cc:2454] T fc37ae55dce445ccbb451cb21d9f03f0 P ddb33c23213d498cbe9a8130ef25827f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b4d7412864047628e61a67a770e9a05 in term 1.
I20260810 23:25:21.731345 18388 leader_election.cc:302] T fc37ae55dce445ccbb451cb21d9f03f0 P 6b4d7412864047628e61a67a770e9a05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b4d7412864047628e61a67a770e9a05, ddb33c23213d498cbe9a8130ef25827f; no voters:
I20260810 23:25:21.731773 18722 raft_consensus.cc:2781] T fc37ae55dce445ccbb451cb21d9f03f0 P 6b4d7412864047628e61a67a770e9a05 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:25:21.731941 18722 raft_consensus.cc:686] T fc37ae55dce445ccbb451cb21d9f03f0 P 6b4d7412864047628e61a67a770e9a05 [term 1 LEADER]: Becoming Leader. State: Replica: 6b4d7412864047628e61a67a770e9a05, State: Running, Role: LEADER
I20260810 23:25:21.732271 18722 consensus_queue.cc:237] T fc37ae55dce445ccbb451cb21d9f03f0 P 6b4d7412864047628e61a67a770e9a05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:21.735438 17819 catalog_manager.cc:5495] T fc37ae55dce445ccbb451cb21d9f03f0 P 6b4d7412864047628e61a67a770e9a05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b4d7412864047628e61a67a770e9a05 (127.14.175.67). New cstate: current_term: 1 leader_uuid: "6b4d7412864047628e61a67a770e9a05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } health_report { overall_health: HEALTHY } } }
I20260810 23:25:21.768276 18801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:21.762679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49966 (local address 127.14.175.67:43999)
0810 23:25:21.763281 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:21.763285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:21.763306 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:25:21.763366 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:21.763370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:21.763432 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:21.763515 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:25:21.763966 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:21.764829 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:21.766381 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:21.766831 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:21.766937 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:21.767503 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:25:21.767671 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:25:21.767743 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:21.767925 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 23:25:21.768002 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":317,"threads_started":1}
I20260810 23:25:21.778961 18287 raft_consensus.cc:1261] T fc37ae55dce445ccbb451cb21d9f03f0 P ddb33c23213d498cbe9a8130ef25827f [term 1 FOLLOWER]: Refusing update from remote peer 6b4d7412864047628e61a67a770e9a05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:25:21.779109 18075 raft_consensus.cc:1261] T fc37ae55dce445ccbb451cb21d9f03f0 P deadca232e58457395f6519bdc10494e [term 1 FOLLOWER]: Refusing update from remote peer 6b4d7412864047628e61a67a770e9a05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:25:21.779574 18747 consensus_queue.cc:1035] T fc37ae55dce445ccbb451cb21d9f03f0 P 6b4d7412864047628e61a67a770e9a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:25:21.779876 18722 consensus_queue.cc:1035] T fc37ae55dce445ccbb451cb21d9f03f0 P 6b4d7412864047628e61a67a770e9a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:25:21.837561 17819 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:25:21.837885 17819 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:25:21.841408 17819 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8420095ccbb64e43b12d94041d536850: Sending DeleteTablet for 3 replicas of tablet fc37ae55dce445ccbb451cb21d9f03f0
I20260810 23:25:21.842307 18042 tablet_service.cc:1514] Processing DeleteTablet for tablet fc37ae55dce445ccbb451cb21d9f03f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:25:21 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:37362
I20260810 23:25:21.842365 18250 tablet_service.cc:1514] Processing DeleteTablet for tablet fc37ae55dce445ccbb451cb21d9f03f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:25:21 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:52048
I20260810 23:25:21.842433 18469 tablet_service.cc:1514] Processing DeleteTablet for tablet fc37ae55dce445ccbb451cb21d9f03f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:25:21 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:40504
I20260810 23:25:21.842676 18760 tablet_replica.cc:330] T fc37ae55dce445ccbb451cb21d9f03f0 P deadca232e58457395f6519bdc10494e: stopping tablet replica
I20260810 23:25:21.842724 18765 tablet_replica.cc:330] T fc37ae55dce445ccbb451cb21d9f03f0 P ddb33c23213d498cbe9a8130ef25827f: stopping tablet replica
I20260810 23:25:21.842905 18763 tablet_replica.cc:330] T fc37ae55dce445ccbb451cb21d9f03f0 P 6b4d7412864047628e61a67a770e9a05: stopping tablet replica
I20260810 23:25:21.842942 18765 raft_consensus.cc:2229] T fc37ae55dce445ccbb451cb21d9f03f0 P ddb33c23213d498cbe9a8130ef25827f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:25:21.843007 18760 raft_consensus.cc:2229] T fc37ae55dce445ccbb451cb21d9f03f0 P deadca232e58457395f6519bdc10494e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:25:21.843124 18763 raft_consensus.cc:2229] T fc37ae55dce445ccbb451cb21d9f03f0 P 6b4d7412864047628e61a67a770e9a05 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:25:21.843142 18765 raft_consensus.cc:2258] T fc37ae55dce445ccbb451cb21d9f03f0 P ddb33c23213d498cbe9a8130ef25827f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:25:21.843268 18760 raft_consensus.cc:2258] T fc37ae55dce445ccbb451cb21d9f03f0 P deadca232e58457395f6519bdc10494e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:25:21.843454 18763 raft_consensus.cc:2258] T fc37ae55dce445ccbb451cb21d9f03f0 P 6b4d7412864047628e61a67a770e9a05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:25:21.843827 17819 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:25:21.844550 18763 ts_tablet_manager.cc:1905] T fc37ae55dce445ccbb451cb21d9f03f0 P 6b4d7412864047628e61a67a770e9a05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:25:21.844550 18765 ts_tablet_manager.cc:1905] T fc37ae55dce445ccbb451cb21d9f03f0 P ddb33c23213d498cbe9a8130ef25827f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:25:21.844646 18760 ts_tablet_manager.cc:1905] T fc37ae55dce445ccbb451cb21d9f03f0 P deadca232e58457395f6519bdc10494e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:25:21.844846 17819 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:25:21.852216 18765 ts_tablet_manager.cc:1918] T fc37ae55dce445ccbb451cb21d9f03f0 P ddb33c23213d498cbe9a8130ef25827f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:25:21.852334 18760 ts_tablet_manager.cc:1918] T fc37ae55dce445ccbb451cb21d9f03f0 P deadca232e58457395f6519bdc10494e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:25:21.852510 18765 log.cc:1196] T fc37ae55dce445ccbb451cb21d9f03f0 P ddb33c23213d498cbe9a8130ef25827f: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-1/wal/wals/fc37ae55dce445ccbb451cb21d9f03f0
I20260810 23:25:21.852517 18763 ts_tablet_manager.cc:1918] T fc37ae55dce445ccbb451cb21d9f03f0 P 6b4d7412864047628e61a67a770e9a05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:25:21.852778 18763 log.cc:1196] T fc37ae55dce445ccbb451cb21d9f03f0 P 6b4d7412864047628e61a67a770e9a05: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-2/wal/wals/fc37ae55dce445ccbb451cb21d9f03f0
I20260810 23:25:21.852814 18760 log.cc:1196] T fc37ae55dce445ccbb451cb21d9f03f0 P deadca232e58457395f6519bdc10494e: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-0/wal/wals/fc37ae55dce445ccbb451cb21d9f03f0
I20260810 23:25:21.853331 18765 ts_tablet_manager.cc:1939] T fc37ae55dce445ccbb451cb21d9f03f0 P ddb33c23213d498cbe9a8130ef25827f: Deleting consensus metadata
I20260810 23:25:21.853422 18760 ts_tablet_manager.cc:1939] T fc37ae55dce445ccbb451cb21d9f03f0 P deadca232e58457395f6519bdc10494e: Deleting consensus metadata
I20260810 23:25:21.854081 18042 tablet_service.cc:1467] Processing CreateTablet for tablet 8f54d61d01b0427a8d531c9a33f1b1a3 (DEFAULT_TABLE table=test-table [id=22956d69ed474197982edd8d3e831dfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:25:21.853979 18469 tablet_service.cc:1467] Processing CreateTablet for tablet 8f54d61d01b0427a8d531c9a33f1b1a3 (DEFAULT_TABLE table=test-table [id=22956d69ed474197982edd8d3e831dfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:25:21.854604 18042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f54d61d01b0427a8d531c9a33f1b1a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:21.854653 18469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f54d61d01b0427a8d531c9a33f1b1a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:21.854868 17793 catalog_manager.cc:4841] TS deadca232e58457395f6519bdc10494e (127.14.175.65:44481): tablet fc37ae55dce445ccbb451cb21d9f03f0 (table test-table [id=c4c40292fd9e43a89894eff13434032c]) successfully deleted
I20260810 23:25:21.855125 17793 catalog_manager.cc:4841] TS ddb33c23213d498cbe9a8130ef25827f (127.14.175.66:45839): tablet fc37ae55dce445ccbb451cb21d9f03f0 (table test-table [id=c4c40292fd9e43a89894eff13434032c]) successfully deleted
I20260810 23:25:21.855090 18250 tablet_service.cc:1467] Processing CreateTablet for tablet 8f54d61d01b0427a8d531c9a33f1b1a3 (DEFAULT_TABLE table=test-table [id=22956d69ed474197982edd8d3e831dfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:25:21.855581 18250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f54d61d01b0427a8d531c9a33f1b1a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:21.855916 18763 ts_tablet_manager.cc:1939] T fc37ae55dce445ccbb451cb21d9f03f0 P 6b4d7412864047628e61a67a770e9a05: Deleting consensus metadata
I20260810 23:25:21.858526 17789 catalog_manager.cc:4841] TS 6b4d7412864047628e61a67a770e9a05 (127.14.175.67:43999): tablet fc37ae55dce445ccbb451cb21d9f03f0 (table test-table [id=c4c40292fd9e43a89894eff13434032c]) successfully deleted
I20260810 23:25:21.862031 18782 tablet_bootstrap.cc:492] T 8f54d61d01b0427a8d531c9a33f1b1a3 P 6b4d7412864047628e61a67a770e9a05: Bootstrap starting.
I20260810 23:25:21.862259 18785 tablet_bootstrap.cc:492] T 8f54d61d01b0427a8d531c9a33f1b1a3 P deadca232e58457395f6519bdc10494e: Bootstrap starting.
I20260810 23:25:21.864032 18782 tablet_bootstrap.cc:654] T 8f54d61d01b0427a8d531c9a33f1b1a3 P 6b4d7412864047628e61a67a770e9a05: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:21.864051 18785 tablet_bootstrap.cc:654] T 8f54d61d01b0427a8d531c9a33f1b1a3 P deadca232e58457395f6519bdc10494e: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:21.864858 18787 tablet_bootstrap.cc:492] T 8f54d61d01b0427a8d531c9a33f1b1a3 P ddb33c23213d498cbe9a8130ef25827f: Bootstrap starting.
I20260810 23:25:21.866632 18787 tablet_bootstrap.cc:654] T 8f54d61d01b0427a8d531c9a33f1b1a3 P ddb33c23213d498cbe9a8130ef25827f: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:21.866710 18785 tablet_bootstrap.cc:492] T 8f54d61d01b0427a8d531c9a33f1b1a3 P deadca232e58457395f6519bdc10494e: No bootstrap required, opened a new log
I20260810 23:25:21.866892 18785 ts_tablet_manager.cc:1397] T 8f54d61d01b0427a8d531c9a33f1b1a3 P deadca232e58457395f6519bdc10494e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260810 23:25:21.866953 18782 tablet_bootstrap.cc:492] T 8f54d61d01b0427a8d531c9a33f1b1a3 P 6b4d7412864047628e61a67a770e9a05: No bootstrap required, opened a new log
I20260810 23:25:21.867161 18782 ts_tablet_manager.cc:1397] T 8f54d61d01b0427a8d531c9a33f1b1a3 P 6b4d7412864047628e61a67a770e9a05: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260810 23:25:21.867763 18785 raft_consensus.cc:348] T 8f54d61d01b0427a8d531c9a33f1b1a3 P deadca232e58457395f6519bdc10494e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:21.868172 18785 raft_consensus.cc:374] T 8f54d61d01b0427a8d531c9a33f1b1a3 P deadca232e58457395f6519bdc10494e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:21.868086 18782 raft_consensus.cc:348] T 8f54d61d01b0427a8d531c9a33f1b1a3 P 6b4d7412864047628e61a67a770e9a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:21.868345 18785 raft_consensus.cc:729] T 8f54d61d01b0427a8d531c9a33f1b1a3 P deadca232e58457395f6519bdc10494e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deadca232e58457395f6519bdc10494e, State: Initialized, Role: FOLLOWER
I20260810 23:25:21.868360 18782 raft_consensus.cc:374] T 8f54d61d01b0427a8d531c9a33f1b1a3 P 6b4d7412864047628e61a67a770e9a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:21.868441 18782 raft_consensus.cc:729] T 8f54d61d01b0427a8d531c9a33f1b1a3 P 6b4d7412864047628e61a67a770e9a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b4d7412864047628e61a67a770e9a05, State: Initialized, Role: FOLLOWER
I20260810 23:25:21.868690 18782 consensus_queue.cc:260] T 8f54d61d01b0427a8d531c9a33f1b1a3 P 6b4d7412864047628e61a67a770e9a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:21.868752 18785 consensus_queue.cc:260] T 8f54d61d01b0427a8d531c9a33f1b1a3 P deadca232e58457395f6519bdc10494e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:21.869396 18782 ts_tablet_manager.cc:1428] T 8f54d61d01b0427a8d531c9a33f1b1a3 P 6b4d7412864047628e61a67a770e9a05: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:25:21.870211 18785 ts_tablet_manager.cc:1428] T 8f54d61d01b0427a8d531c9a33f1b1a3 P deadca232e58457395f6519bdc10494e: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260810 23:25:21.871491 18787 tablet_bootstrap.cc:492] T 8f54d61d01b0427a8d531c9a33f1b1a3 P ddb33c23213d498cbe9a8130ef25827f: No bootstrap required, opened a new log
I20260810 23:25:21.871837 18787 ts_tablet_manager.cc:1397] T 8f54d61d01b0427a8d531c9a33f1b1a3 P ddb33c23213d498cbe9a8130ef25827f: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 23:25:21.873195 18787 raft_consensus.cc:348] T 8f54d61d01b0427a8d531c9a33f1b1a3 P ddb33c23213d498cbe9a8130ef25827f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:21.873539 18787 raft_consensus.cc:374] T 8f54d61d01b0427a8d531c9a33f1b1a3 P ddb33c23213d498cbe9a8130ef25827f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:21.873652 18787 raft_consensus.cc:729] T 8f54d61d01b0427a8d531c9a33f1b1a3 P ddb33c23213d498cbe9a8130ef25827f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddb33c23213d498cbe9a8130ef25827f, State: Initialized, Role: FOLLOWER
I20260810 23:25:21.873971 18787 consensus_queue.cc:260] T 8f54d61d01b0427a8d531c9a33f1b1a3 P ddb33c23213d498cbe9a8130ef25827f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:21.874593 18787 ts_tablet_manager.cc:1428] T 8f54d61d01b0427a8d531c9a33f1b1a3 P ddb33c23213d498cbe9a8130ef25827f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260810 23:25:22.048143 18793 raft_consensus.cc:482] T 8f54d61d01b0427a8d531c9a33f1b1a3 P deadca232e58457395f6519bdc10494e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:25:22.048570 18793 raft_consensus.cc:504] T 8f54d61d01b0427a8d531c9a33f1b1a3 P deadca232e58457395f6519bdc10494e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:22.050457 18793 leader_election.cc:288] T 8f54d61d01b0427a8d531c9a33f1b1a3 P deadca232e58457395f6519bdc10494e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddb33c23213d498cbe9a8130ef25827f (127.14.175.66:45839), 6b4d7412864047628e61a67a770e9a05 (127.14.175.67:43999)
I20260810 23:25:22.058583 18801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:22.050858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:49345 (local address 127.14.175.67:43999)
0810 23:25:22.051058 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:22.051065 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:22.052514 (+ 1449us) server_negotiation.cc:407] Connection header received
0810 23:25:22.052732 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:22.052736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:22.052800 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:22.052876 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:25:22.053766 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:22.054589 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:22.056623 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:22.057571 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:22.057750 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:22.057831 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:22.058148 (+ 317us) server_negotiation.cc:299] Negotiation successful
0810 23:25:22.058275 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 23:25:22.059078 18814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:22.050767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:43999 (local address 127.14.175.65:49345)
0810 23:25:22.052382 (+ 1615us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:22.052400 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:22.052573 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:22.052925 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:22.052930 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:22.052967 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:25:22.053509 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:22.053523 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:22.054740 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:22.054745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:22.056425 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:22.056439 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:22.057806 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:22.057814 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:22.057983 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:22.058011 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 23:25:22.058100 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 23:25:22.058240 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1487,"spinlock_wait_cycles":9472,"thread_start_us":1234,"threads_started":1}
I20260810 23:25:22.059970 18507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f54d61d01b0427a8d531c9a33f1b1a3" candidate_uuid: "deadca232e58457395f6519bdc10494e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b4d7412864047628e61a67a770e9a05" is_pre_election: true
I20260810 23:25:22.060303 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:22.051005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:45839 (local address 127.14.175.65:58783)
0810 23:25:22.052910 (+ 1905us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:22.052925 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:22.053104 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:22.053463 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:22.053468 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:22.053491 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:25:22.054089 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:22.054101 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:22.055652 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:22.055656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:22.057505 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:22.057523 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:22.059608 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:22.059614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:22.059783 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:22.059809 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 23:25:22.059887 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 23:25:22.059988 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1738,"thread_start_us":1540,"threads_started":1}
I20260810 23:25:22.060312 18507 raft_consensus.cc:2454] T 8f54d61d01b0427a8d531c9a33f1b1a3 P 6b4d7412864047628e61a67a770e9a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate deadca232e58457395f6519bdc10494e in term 0.
I20260810 23:25:22.060587 18815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:22.051036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:58783 (local address 127.14.175.66:45839)
0810 23:25:22.052249 (+ 1213us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:22.052256 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:22.053036 (+ 780us) server_negotiation.cc:407] Connection header received
0810 23:25:22.053261 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:22.053266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:22.053329 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:22.053414 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:25:22.054327 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:22.055499 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:22.057691 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:22.059318 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:22.059494 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:22.059567 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:22.059950 (+ 383us) server_negotiation.cc:299] Negotiation successful
0810 23:25:22.060148 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":961,"thread_start_us":654,"threads_started":1}
I20260810 23:25:22.060930 17960 leader_election.cc:302] T 8f54d61d01b0427a8d531c9a33f1b1a3 P deadca232e58457395f6519bdc10494e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b4d7412864047628e61a67a770e9a05, deadca232e58457395f6519bdc10494e; no voters:
I20260810 23:25:22.061239 18287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f54d61d01b0427a8d531c9a33f1b1a3" candidate_uuid: "deadca232e58457395f6519bdc10494e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb33c23213d498cbe9a8130ef25827f" is_pre_election: true
I20260810 23:25:22.061517 18287 raft_consensus.cc:2454] T 8f54d61d01b0427a8d531c9a33f1b1a3 P ddb33c23213d498cbe9a8130ef25827f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate deadca232e58457395f6519bdc10494e in term 0.
I20260810 23:25:22.061616 18793 raft_consensus.cc:2781] T 8f54d61d01b0427a8d531c9a33f1b1a3 P deadca232e58457395f6519bdc10494e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:25:22.061826 18793 raft_consensus.cc:482] T 8f54d61d01b0427a8d531c9a33f1b1a3 P deadca232e58457395f6519bdc10494e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:25:22.061932 18793 raft_consensus.cc:3037] T 8f54d61d01b0427a8d531c9a33f1b1a3 P deadca232e58457395f6519bdc10494e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:22.065011 18793 raft_consensus.cc:504] T 8f54d61d01b0427a8d531c9a33f1b1a3 P deadca232e58457395f6519bdc10494e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:22.065692 18793 leader_election.cc:288] T 8f54d61d01b0427a8d531c9a33f1b1a3 P deadca232e58457395f6519bdc10494e [CANDIDATE]: Term 1 election: Requested vote from peers ddb33c23213d498cbe9a8130ef25827f (127.14.175.66:45839), 6b4d7412864047628e61a67a770e9a05 (127.14.175.67:43999)
I20260810 23:25:22.066303 18507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f54d61d01b0427a8d531c9a33f1b1a3" candidate_uuid: "deadca232e58457395f6519bdc10494e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b4d7412864047628e61a67a770e9a05"
I20260810 23:25:22.066293 18287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f54d61d01b0427a8d531c9a33f1b1a3" candidate_uuid: "deadca232e58457395f6519bdc10494e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb33c23213d498cbe9a8130ef25827f"
I20260810 23:25:22.066581 18507 raft_consensus.cc:3037] T 8f54d61d01b0427a8d531c9a33f1b1a3 P 6b4d7412864047628e61a67a770e9a05 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:22.066694 18287 raft_consensus.cc:3037] T 8f54d61d01b0427a8d531c9a33f1b1a3 P ddb33c23213d498cbe9a8130ef25827f [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:22.069070 18507 raft_consensus.cc:2454] T 8f54d61d01b0427a8d531c9a33f1b1a3 P 6b4d7412864047628e61a67a770e9a05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate deadca232e58457395f6519bdc10494e in term 1.
I20260810 23:25:22.069252 18287 raft_consensus.cc:2454] T 8f54d61d01b0427a8d531c9a33f1b1a3 P ddb33c23213d498cbe9a8130ef25827f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate deadca232e58457395f6519bdc10494e in term 1.
I20260810 23:25:22.069773 17963 leader_election.cc:302] T 8f54d61d01b0427a8d531c9a33f1b1a3 P deadca232e58457395f6519bdc10494e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6b4d7412864047628e61a67a770e9a05, ddb33c23213d498cbe9a8130ef25827f, deadca232e58457395f6519bdc10494e; no voters:
I20260810 23:25:22.070341 18793 raft_consensus.cc:2781] T 8f54d61d01b0427a8d531c9a33f1b1a3 P deadca232e58457395f6519bdc10494e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:25:22.071339 18793 raft_consensus.cc:686] T 8f54d61d01b0427a8d531c9a33f1b1a3 P deadca232e58457395f6519bdc10494e [term 1 LEADER]: Becoming Leader. State: Replica: deadca232e58457395f6519bdc10494e, State: Running, Role: LEADER
I20260810 23:25:22.071832 18793 consensus_queue.cc:237] T 8f54d61d01b0427a8d531c9a33f1b1a3 P deadca232e58457395f6519bdc10494e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } }
I20260810 23:25:22.076030 17819 catalog_manager.cc:5495] T 8f54d61d01b0427a8d531c9a33f1b1a3 P deadca232e58457395f6519bdc10494e reported cstate change: term changed from 0 to 1, leader changed from <none> to deadca232e58457395f6519bdc10494e (127.14.175.65). New cstate: current_term: 1 leader_uuid: "deadca232e58457395f6519bdc10494e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadca232e58457395f6519bdc10494e" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 44481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 } health_report { overall_health: UNKNOWN } } }
I20260810 23:25:22.128931 18821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:22.121139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44910 (local address 127.14.175.65:44481)
0810 23:25:22.122073 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:22.122078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:22.122099 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:25:22.122211 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:22.122216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:22.122308 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:22.122476 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:25:22.123152 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:22.124147 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:22.125910 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:22.126644 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:22.126757 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:22.127501 (+ 744us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:25:22.128170 (+ 669us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:25:22.128298 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:22.128352 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 23:25:22.128428 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":789,"thread_start_us":541,"threads_started":1}
I20260810 23:25:22.149544 18815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:22.142124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:41415 (local address 127.14.175.66:45839)
0810 23:25:22.142355 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:22.142363 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:22.143513 (+ 1150us) server_negotiation.cc:407] Connection header received
0810 23:25:22.143745 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:22.143749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:22.143822 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:22.143893 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:25:22.144824 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:22.145854 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:22.147447 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:22.148016 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:22.148121 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:22.148728 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:25:22.148850 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:25:22.148920 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:22.149056 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 23:25:22.149191 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 23:25:22.149690 18827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:22.142108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:45839 (local address 127.14.175.65:41415)
0810 23:25:22.143406 (+ 1298us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:22.143425 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:22.143608 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:22.143957 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:22.143961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:22.143972 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:25:22.144648 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:22.144663 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:22.145992 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:22.145996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:22.147301 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:22.147315 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:22.148259 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:22.148265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:22.148438 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:22.148507 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:25:22.148927 (+ 420us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:25:22.148935 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 23:25:22.149000 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 23:25:22.149130 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":961,"thread_start_us":613,"threads_started":1}
I20260810 23:25:22.158387 18507 raft_consensus.cc:1261] T 8f54d61d01b0427a8d531c9a33f1b1a3 P 6b4d7412864047628e61a67a770e9a05 [term 1 FOLLOWER]: Refusing update from remote peer deadca232e58457395f6519bdc10494e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:25:22.158702 18287 raft_consensus.cc:1261] T 8f54d61d01b0427a8d531c9a33f1b1a3 P ddb33c23213d498cbe9a8130ef25827f [term 1 FOLLOWER]: Refusing update from remote peer deadca232e58457395f6519bdc10494e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:25:22.159219 18817 consensus_queue.cc:1035] T 8f54d61d01b0427a8d531c9a33f1b1a3 P deadca232e58457395f6519bdc10494e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b4d7412864047628e61a67a770e9a05" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:25:22.159855 18793 consensus_queue.cc:1035] T 8f54d61d01b0427a8d531c9a33f1b1a3 P deadca232e58457395f6519bdc10494e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb33c23213d498cbe9a8130ef25827f" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 45839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:25:22.188377 18821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:22.182274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:37135 (local address 127.14.175.65:44481)
0810 23:25:22.182719 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:22.182725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:22.182835 (+ 110us) server_negotiation.cc:407] Connection header received
0810 23:25:22.183015 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:22.183020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:22.183091 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:22.183166 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:25:22.183783 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:22.184597 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:22.186707 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:22.187222 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:22.187326 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:22.187782 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:25:22.187889 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:25:22.187956 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:22.188091 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 23:25:22.188161 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328}
I20260810 23:25:22.188575 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:22.182009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:44481 (local address 127.14.175.66:37135)
0810 23:25:22.182738 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:22.182757 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:22.182892 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:22.183196 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:22.183200 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:22.183212 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:25:22.183559 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:22.183571 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:22.184740 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:22.184744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:22.186564 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:22.186581 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:22.187434 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:22.187439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:22.187574 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:22.187623 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:25:22.187990 (+ 367us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:25:22.188001 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 23:25:22.188063 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 23:25:22.188184 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":435,"threads_started":1}
W20260810 23:25:22.190130 18734 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 23:25:23.801950 17817 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:25:23.802256 17817 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:25:23.807098 17817 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8420095ccbb64e43b12d94041d536850: Sending DeleteTablet for 3 replicas of tablet 8f54d61d01b0427a8d531c9a33f1b1a3
I20260810 23:25:23.808012 18042 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f54d61d01b0427a8d531c9a33f1b1a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:25:23 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:37362
I20260810 23:25:23.808243 18469 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f54d61d01b0427a8d531c9a33f1b1a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:25:23 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:40504
I20260810 23:25:23.808233 18250 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f54d61d01b0427a8d531c9a33f1b1a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:25:23 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:52048
I20260810 23:25:23.811655 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 17939
I20260810 23:25:23.819954 18871 tablet_replica.cc:330] T 8f54d61d01b0427a8d531c9a33f1b1a3 P 6b4d7412864047628e61a67a770e9a05: stopping tablet replica
I20260810 23:25:23.821559 18871 raft_consensus.cc:2229] T 8f54d61d01b0427a8d531c9a33f1b1a3 P 6b4d7412864047628e61a67a770e9a05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:25:23.822358 18871 raft_consensus.cc:2258] T 8f54d61d01b0427a8d531c9a33f1b1a3 P 6b4d7412864047628e61a67a770e9a05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:25:23.824573 18871 ts_tablet_manager.cc:1905] T 8f54d61d01b0427a8d531c9a33f1b1a3 P 6b4d7412864047628e61a67a770e9a05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:25:23.825659 18869 tablet_replica.cc:330] T 8f54d61d01b0427a8d531c9a33f1b1a3 P deadca232e58457395f6519bdc10494e: stopping tablet replica
I20260810 23:25:23.825949 18869 raft_consensus.cc:2229] T 8f54d61d01b0427a8d531c9a33f1b1a3 P deadca232e58457395f6519bdc10494e [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:25:23.826627 18869 raft_consensus.cc:2258] T 8f54d61d01b0427a8d531c9a33f1b1a3 P deadca232e58457395f6519bdc10494e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:25:23.830380 18869 ts_tablet_manager.cc:1905] T 8f54d61d01b0427a8d531c9a33f1b1a3 P deadca232e58457395f6519bdc10494e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:25:23.832577 18871 ts_tablet_manager.cc:1918] T 8f54d61d01b0427a8d531c9a33f1b1a3 P 6b4d7412864047628e61a67a770e9a05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
Aug 10 23:25:23 dist-test-slave-rmhg krb5kdc[17740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404321, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.175.65@KRBTEST.COM
I20260810 23:25:23.833237 18873 tablet_replica.cc:330] T 8f54d61d01b0427a8d531c9a33f1b1a3 P ddb33c23213d498cbe9a8130ef25827f: stopping tablet replica
I20260810 23:25:23.833454 18873 raft_consensus.cc:2229] T 8f54d61d01b0427a8d531c9a33f1b1a3 P ddb33c23213d498cbe9a8130ef25827f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:25:23.833724 18873 raft_consensus.cc:2258] T 8f54d61d01b0427a8d531c9a33f1b1a3 P ddb33c23213d498cbe9a8130ef25827f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:25:23.834769 18873 ts_tablet_manager.cc:1905] T 8f54d61d01b0427a8d531c9a33f1b1a3 P ddb33c23213d498cbe9a8130ef25827f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:25:23.834010 18871 log.cc:1196] T 8f54d61d01b0427a8d531c9a33f1b1a3 P 6b4d7412864047628e61a67a770e9a05: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-2/wal/wals/8f54d61d01b0427a8d531c9a33f1b1a3
I20260810 23:25:23.835474 18871 ts_tablet_manager.cc:1939] T 8f54d61d01b0427a8d531c9a33f1b1a3 P 6b4d7412864047628e61a67a770e9a05: Deleting consensus metadata
I20260810 23:25:23.836819 17789 catalog_manager.cc:4841] TS 6b4d7412864047628e61a67a770e9a05 (127.14.175.67:43999): tablet 8f54d61d01b0427a8d531c9a33f1b1a3 (table test-table [id=22956d69ed474197982edd8d3e831dfe]) successfully deleted
I20260810 23:25:23.839553 18869 ts_tablet_manager.cc:1918] T 8f54d61d01b0427a8d531c9a33f1b1a3 P deadca232e58457395f6519bdc10494e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 23:25:23.839702 18869 log.cc:1196] T 8f54d61d01b0427a8d531c9a33f1b1a3 P deadca232e58457395f6519bdc10494e: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-0/wal/wals/8f54d61d01b0427a8d531c9a33f1b1a3
I20260810 23:25:23.840126 18869 ts_tablet_manager.cc:1939] T 8f54d61d01b0427a8d531c9a33f1b1a3 P deadca232e58457395f6519bdc10494e: Deleting consensus metadata
I20260810 23:25:23.841382 17793 catalog_manager.cc:4841] TS deadca232e58457395f6519bdc10494e (127.14.175.65:44481): tablet 8f54d61d01b0427a8d531c9a33f1b1a3 (table test-table [id=22956d69ed474197982edd8d3e831dfe]) successfully deleted
I20260810 23:25:23.841617 18873 ts_tablet_manager.cc:1918] T 8f54d61d01b0427a8d531c9a33f1b1a3 P ddb33c23213d498cbe9a8130ef25827f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 23:25:23.841779 18873 log.cc:1196] T 8f54d61d01b0427a8d531c9a33f1b1a3 P ddb33c23213d498cbe9a8130ef25827f: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404299514076-15037-0/minicluster-data/ts-1/wal/wals/8f54d61d01b0427a8d531c9a33f1b1a3
I20260810 23:25:23.841794 18875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:23.812779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44924 (local address 127.14.175.65:44481)
0810 23:25:23.818598 (+ 5819us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:23.818604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:23.820727 (+ 2123us) server_negotiation.cc:407] Connection header received
0810 23:25:23.820895 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:23.820901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:23.821008 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:23.821196 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:23.823470 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:23.824739 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:23.826972 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:23.827744 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:23.827861 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:23.834461 (+ 6600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:23.834481 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:23.834500 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:23.834542 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:23.838343 (+ 3801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:23.839236 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:23.839244 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:23.839246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:23.839408 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:23.839884 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:23.839889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:23.839891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:23.840343 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:23.840645 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:23.841003 (+ 358us) server_negotiation.cc:299] Negotiation successful
0810 23:25:23.841282 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5660,"thread_start_us":5500,"threads_started":1}
I20260810 23:25:23.842193 18873 ts_tablet_manager.cc:1939] T 8f54d61d01b0427a8d531c9a33f1b1a3 P ddb33c23213d498cbe9a8130ef25827f: Deleting consensus metadata
W20260810 23:25:23.842502 18139 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44924
W20260810 23:25:23.843276 15037 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 17939: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:25:23.843339 17793 catalog_manager.cc:4841] TS ddb33c23213d498cbe9a8130ef25827f (127.14.175.66:45839): tablet 8f54d61d01b0427a8d531c9a33f1b1a3 (table test-table [id=22956d69ed474197982edd8d3e831dfe]) successfully deleted
I20260810 23:25:23.843498 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 17939
I20260810 23:25:23.876844 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 18155
Aug 10 23:25:23 dist-test-slave-rmhg krb5kdc[17740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404321, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.175.66@KRBTEST.COM
I20260810 23:25:23.892731 18876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:23.877625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41358 (local address 127.14.175.66:45839)
0810 23:25:23.878170 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:23.878175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:23.878194 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:25:23.878252 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:23.878256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:23.878319 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:23.878449 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:23.879717 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:23.880493 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:23.881867 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:23.882353 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:23.882460 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:23.886765 (+ 4305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:23.886787 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:23.886806 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:23.886850 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:23.889785 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:23.890584 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:23.890595 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:23.890598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:23.890750 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:23.891107 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:23.891112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:23.891114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:23.891502 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:23.891828 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:23.892153 (+ 325us) server_negotiation.cc:299] Negotiation successful
0810 23:25:23.892362 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":255,"threads_started":1}
W20260810 23:25:23.893481 18340 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41358
W20260810 23:25:23.894162 15037 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 18155: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:25:23.894335 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 18155
I20260810 23:25:23.930439 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 18354
Aug 10 23:25:23 dist-test-slave-rmhg krb5kdc[17740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404321, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.175.67@KRBTEST.COM
I20260810 23:25:23.946861 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:23.931225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49978 (local address 127.14.175.67:43999)
0810 23:25:23.931755 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:23.931760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:23.931776 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:25:23.931833 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:23.931836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:23.931894 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:23.931987 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:23.933481 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:23.934308 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:23.935756 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:23.936226 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:23.936342 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:23.941035 (+ 4693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:23.941054 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:23.941088 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:23.941141 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:23.944051 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:23.944914 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:23.944922 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:23.944924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:23.945092 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:23.945448 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:23.945451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:23.945453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:23.945826 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:23.946060 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:23.946324 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 23:25:23.946514 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":268,"threads_started":1}
W20260810 23:25:23.947566 18578 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49978
W20260810 23:25:23.948278 15037 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 18354: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:25:23.948423 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 18354
I20260810 23:25:23.976661 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 17764
I20260810 23:25:23.978083 17882 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:25:24.097741 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 17764
2026-08-10T23:25:24Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9028 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:25:24 dist-test-slave-rmhg krb5kdc[18881](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:25:24 dist-test-slave-rmhg krb5kdc[18881](info): set up 2 sockets
Aug 10 23:25:24 dist-test-slave-rmhg krb5kdc[18881](info): commencing operation
krb5kdc: starting...
W20260810 23:25:25.331342 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.183s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:25:25 dist-test-slave-rmhg krb5kdc[18881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:25:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:25:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
I20260810 23:25:25.554203 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:36599
--webserver_interface=127.14.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:37047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:36599
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:25.906147 18901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:25.906499 18901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:25.906635 18901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:25.914199 18901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:25:25.914358 18901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:25.914438 18901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:25.914517 18901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:25:25.914580 18901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:25:25.925045 18901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:37047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:36599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:36599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:25.926540 18901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:25.927983 18901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:25.940853 18910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:25.940834 18916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:25.940915 18911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:25.941630 18901 server_base.cc:1034] running on GCE node
I20260810 23:25:25.942560 18901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:25.944216 18901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:25.945457 18901 hybrid_clock.cc:648] HybridClock initialized: now 1786404325945395 us; error 53 us; skew 500 ppm
I20260810 23:25:25.945849 18901 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:25 dist-test-slave-rmhg krb5kdc[18881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404325, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:25.952663 18901 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:25:25.954511 18901 webserver.cc:460] Webserver started at http://127.14.175.126:46347/ using document root <none> and password file <none>
I20260810 23:25:25.955260 18901 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:25.955390 18901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:25.955750 18901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:25.959101 18901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "1508ecf9aac84a3a8e61fd88c674ebcc"
format_stamp: "Formatted at 2026-08-10 23:25:25 on dist-test-slave-rmhg"
server_key: "59a396656eed3b35997b51c1e2731628"
server_key_iv: "96f95c258de93528f6444bafd415b658"
server_key_version: "encryptionkey@0"
I20260810 23:25:25.959937 18901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "1508ecf9aac84a3a8e61fd88c674ebcc"
format_stamp: "Formatted at 2026-08-10 23:25:25 on dist-test-slave-rmhg"
server_key: "59a396656eed3b35997b51c1e2731628"
server_key_iv: "96f95c258de93528f6444bafd415b658"
server_key_version: "encryptionkey@0"
I20260810 23:25:25.965951 18901 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260810 23:25:25.969950 18928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:25.971489 18901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:25:25.971724 18901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "1508ecf9aac84a3a8e61fd88c674ebcc"
format_stamp: "Formatted at 2026-08-10 23:25:25 on dist-test-slave-rmhg"
server_key: "59a396656eed3b35997b51c1e2731628"
server_key_iv: "96f95c258de93528f6444bafd415b658"
server_key_version: "encryptionkey@0"
I20260810 23:25:25.971942 18901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:25.988418 18901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:25.992839 18901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:25.993130 18901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:26.011885 18901 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:36599
I20260810 23:25:26.011904 19012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:36599 every 8 connection(s)
I20260810 23:25:26.013576 18901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:25:26.016937 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 18901
I20260810 23:25:26.017144 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:25:26.017849 15037 external_mini_cluster.cc:1428] Setting key 7389bc4f44c7111fb3517bebc8593c02
I20260810 23:25:26.017958 19013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 23:25:26 dist-test-slave-rmhg krb5kdc[18881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:26.033934 19013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1508ecf9aac84a3a8e61fd88c674ebcc: Bootstrap starting.
I20260810 23:25:26.037900 19013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1508ecf9aac84a3a8e61fd88c674ebcc: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:26.039189 19013 log.cc:824] T 00000000000000000000000000000000 P 1508ecf9aac84a3a8e61fd88c674ebcc: Log is configured to *not* fsync() on all Append() calls
I20260810 23:25:26.042925 19013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1508ecf9aac84a3a8e61fd88c674ebcc: No bootstrap required, opened a new log
I20260810 23:25:26.043545 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:26.019885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53738 (local address 127.14.175.126:36599)
0810 23:25:26.020733 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:26.020751 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:26.020800 (+ 49us) server_negotiation.cc:407] Connection header received
0810 23:25:26.023086 (+ 2286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:26.023117 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:26.023626 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:26.024008 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:26.026148 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:26.027518 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:26.029353 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:26.030110 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:26.030307 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:26.035375 (+ 5068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:26.035385 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:26.035402 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:26.035450 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:26.038732 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:26.039742 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:26.039750 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:26.039758 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:26.040052 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:26.040520 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:26.040525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:26.040528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:26.041229 (+ 701us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:26.041483 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:26.042004 (+ 521us) server_negotiation.cc:299] Negotiation successful
0810 23:25:26.042413 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":392,"threads_started":1}
I20260810 23:25:26.050815 19013 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1508ecf9aac84a3a8e61fd88c674ebcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1508ecf9aac84a3a8e61fd88c674ebcc" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 36599 } }
I20260810 23:25:26.051235 19013 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1508ecf9aac84a3a8e61fd88c674ebcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:26.051347 19013 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1508ecf9aac84a3a8e61fd88c674ebcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1508ecf9aac84a3a8e61fd88c674ebcc, State: Initialized, Role: FOLLOWER
I20260810 23:25:26.051944 19013 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1508ecf9aac84a3a8e61fd88c674ebcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1508ecf9aac84a3a8e61fd88c674ebcc" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 36599 } }
I20260810 23:25:26.052183 19013 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1508ecf9aac84a3a8e61fd88c674ebcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:25:26.052315 19013 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1508ecf9aac84a3a8e61fd88c674ebcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:25:26.052474 19013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1508ecf9aac84a3a8e61fd88c674ebcc [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:26.054806 19013 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1508ecf9aac84a3a8e61fd88c674ebcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1508ecf9aac84a3a8e61fd88c674ebcc" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 36599 } }
I20260810 23:25:26.055338 19013 leader_election.cc:302] T 00000000000000000000000000000000 P 1508ecf9aac84a3a8e61fd88c674ebcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1508ecf9aac84a3a8e61fd88c674ebcc; no voters:
I20260810 23:25:26.055914 19013 leader_election.cc:288] T 00000000000000000000000000000000 P 1508ecf9aac84a3a8e61fd88c674ebcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:25:26.056039 19020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1508ecf9aac84a3a8e61fd88c674ebcc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:25:26.057272 19020 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1508ecf9aac84a3a8e61fd88c674ebcc [term 1 LEADER]: Becoming Leader. State: Replica: 1508ecf9aac84a3a8e61fd88c674ebcc, State: Running, Role: LEADER
I20260810 23:25:26.057840 19020 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1508ecf9aac84a3a8e61fd88c674ebcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1508ecf9aac84a3a8e61fd88c674ebcc" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 36599 } }
I20260810 23:25:26.058229 19013 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1508ecf9aac84a3a8e61fd88c674ebcc [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:25:26.060577 19022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1508ecf9aac84a3a8e61fd88c674ebcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1508ecf9aac84a3a8e61fd88c674ebcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1508ecf9aac84a3a8e61fd88c674ebcc" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 36599 } } }
I20260810 23:25:26.061059 19022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1508ecf9aac84a3a8e61fd88c674ebcc [sys.catalog]: This master's current role is: LEADER
I20260810 23:25:26.061137 19023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1508ecf9aac84a3a8e61fd88c674ebcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1508ecf9aac84a3a8e61fd88c674ebcc. Latest consensus state: current_term: 1 leader_uuid: "1508ecf9aac84a3a8e61fd88c674ebcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1508ecf9aac84a3a8e61fd88c674ebcc" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 36599 } } }
I20260810 23:25:26.061482 19023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1508ecf9aac84a3a8e61fd88c674ebcc [sys.catalog]: This master's current role is: LEADER
I20260810 23:25:26.063422 19033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:25:26.068157 19033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:25:26.077705 19033 catalog_manager.cc:1347] Generated new cluster ID: cc184dde592540e7abf12c4d4b1fffcc
I20260810 23:25:26.077832 19033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:25:26.093659 19033 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:25:26.095687 19033 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:25:26.109086 19033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1508ecf9aac84a3a8e61fd88c674ebcc: Generated new TSK 0
I20260810 23:25:26.110311 19033 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
I20260810 23:25:26.174093 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.65:0
--local_ip_for_outbound_sockets=127.14.175.65
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:36599
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:37047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:26.526293 19051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:26.526618 19051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:26.526742 19051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:26.534327 19051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:26.534458 19051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:26.534622 19051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.65
I20260810 23:25:26.545089 19051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:37047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:36599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.65
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:26.546564 19051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:26.547976 19051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:26.561774 19058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:26.561739 19061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:26.561813 19057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:26.562675 19051 server_base.cc:1034] running on GCE node
I20260810 23:25:26.563436 19051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:26.564472 19051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:26.565702 19051 hybrid_clock.cc:648] HybridClock initialized: now 1786404326565645 us; error 48 us; skew 500 ppm
I20260810 23:25:26.566057 19051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:26 dist-test-slave-rmhg krb5kdc[18881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404326, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:26.569334 19051 init.cc:377] Logged in from keytab as kudu/127.14.175.65@KRBTEST.COM (short username kudu)
I20260810 23:25:26.570930 19051 webserver.cc:460] Webserver started at http://127.14.175.65:46815/ using document root <none> and password file <none>
I20260810 23:25:26.571663 19051 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:26.571784 19051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:26.572109 19051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:26.575273 19051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-0/data/instance:
uuid: "3519fdd7c4b44e3398b586ff48809317"
format_stamp: "Formatted at 2026-08-10 23:25:26 on dist-test-slave-rmhg"
server_key: "887e604151ec94dbe23962182dd4fa52"
server_key_iv: "44ab1391c7bd12a395cb8a6e5099eaa2"
server_key_version: "encryptionkey@0"
I20260810 23:25:26.576174 19051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance:
uuid: "3519fdd7c4b44e3398b586ff48809317"
format_stamp: "Formatted at 2026-08-10 23:25:26 on dist-test-slave-rmhg"
server_key: "887e604151ec94dbe23962182dd4fa52"
server_key_iv: "44ab1391c7bd12a395cb8a6e5099eaa2"
server_key_version: "encryptionkey@0"
I20260810 23:25:26.581938 19051 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 23:25:26.585733 19075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:26.587132 19051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 23:25:26.587301 19051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-0/wal
uuid: "3519fdd7c4b44e3398b586ff48809317"
format_stamp: "Formatted at 2026-08-10 23:25:26 on dist-test-slave-rmhg"
server_key: "887e604151ec94dbe23962182dd4fa52"
server_key_iv: "44ab1391c7bd12a395cb8a6e5099eaa2"
server_key_version: "encryptionkey@0"
I20260810 23:25:26.587502 19051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:26.608320 19051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:26.613271 19051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:26.613576 19051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:26.614616 19051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:25:26.616066 19051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:25:26.616155 19051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:26.616297 19051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:25:26.616353 19051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:26.648674 19051 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.65:45851
I20260810 23:25:26.648693 19242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.65:45851 every 8 connection(s)
I20260810 23:25:26.650174 19051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
I20260810 23:25:26.650774 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 19051
I20260810 23:25:26.650970 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance
I20260810 23:25:26.651799 15037 external_mini_cluster.cc:1428] Setting key a2544a6b7bc6bef1c813483207fed078
Aug 10 23:25:26 dist-test-slave-rmhg krb5kdc[18881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404326, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.175.66@KRBTEST.COM; defaulting to no policy
I20260810 23:25:26.672740 19248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:26.653002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:46949 (local address 127.14.175.126:36599)
0810 23:25:26.653528 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:26.653534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:26.654676 (+ 1142us) server_negotiation.cc:407] Connection header received
0810 23:25:26.656598 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:26.656604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:26.656690 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:26.656788 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:26.658580 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:26.659395 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:26.661048 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:26.661518 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:26.661650 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:26.665711 (+ 4061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:26.665718 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:26.665723 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:26.665766 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:26.669116 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:26.670073 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:26.670079 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:26.670083 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:26.670253 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:26.670784 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:26.670789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:26.670792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:26.671186 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:26.671346 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:26.672145 (+ 799us) server_negotiation.cc:299] Negotiation successful
0810 23:25:26.672409 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":273,"threads_started":1}
I20260810 23:25:26.673730 19249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:26.653711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:36599 (local address 127.14.175.65:46949)
0810 23:25:26.654502 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:26.654555 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:26.656259 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:26.657056 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:26.657072 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:26.657705 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:26.658380 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:26.658410 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:26.659583 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:26.659589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:26.660895 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:26.660908 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:26.661779 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:26.661786 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:26.662085 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:26.662788 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:26.662818 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:26.665397 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:26.669325 (+ 3928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:26.669334 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:26.669365 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:26.669920 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:26.670388 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:26.670392 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:26.670394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:26.670656 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:26.671343 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:26.671351 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:26.671734 (+ 383us) client_negotiation.cc:769] Sending connection context
0810 23:25:26.671982 (+ 248us) client_negotiation.cc:240] Negotiation successful
0810 23:25:26.672466 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":319,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.66@KRBTEST.COM" created.
I20260810 23:25:26.675518 19243 heartbeater.cc:347] Connected to a master server at 127.14.175.126:36599
I20260810 23:25:26.675859 19243 heartbeater.cc:464] Registering TS with master...
I20260810 23:25:26.676694 19243 heartbeater.cc:511] Master 127.14.175.126:36599 requested a full tablet report, sending...
I20260810 23:25:26.679319 18959 ts_manager.cc:194] Registered new tserver with Master: 3519fdd7c4b44e3398b586ff48809317 (127.14.175.65:45851)
I20260810 23:25:26.681313 18959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.65@KRBTEST.COM'} at 127.14.175.65:46949
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
I20260810 23:25:26.702765 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.66:0
--local_ip_for_outbound_sockets=127.14.175.66
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:36599
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:37047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:27.047053 19253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:27.047377 19253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:27.047523 19253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:27.055063 19253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:27.055207 19253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:27.055358 19253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.66
I20260810 23:25:27.065893 19253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:37047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:36599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.66
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:27.067299 19253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:27.068693 19253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:27.082297 19259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:27.082338 19261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:27.082343 19264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:27.083519 19253 server_base.cc:1034] running on GCE node
I20260810 23:25:27.084188 19253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:27.085245 19253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:27.086452 19253 hybrid_clock.cc:648] HybridClock initialized: now 1786404327086387 us; error 61 us; skew 500 ppm
I20260810 23:25:27.086781 19253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:27 dist-test-slave-rmhg krb5kdc[18881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404327, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:27.090258 19253 init.cc:377] Logged in from keytab as kudu/127.14.175.66@KRBTEST.COM (short username kudu)
I20260810 23:25:27.092195 19253 webserver.cc:460] Webserver started at http://127.14.175.66:33929/ using document root <none> and password file <none>
I20260810 23:25:27.092998 19253 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:27.093135 19253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:27.093559 19253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:27.096832 19253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-1/data/instance:
uuid: "5c629c8bc36e4d4893d30f9a7c91f869"
format_stamp: "Formatted at 2026-08-10 23:25:27 on dist-test-slave-rmhg"
server_key: "2dc86f7fafb68c62f18c2ae0316672ed"
server_key_iv: "62a7811f985cb76ad12255c91d6edd3a"
server_key_version: "encryptionkey@0"
I20260810 23:25:27.097769 19253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance:
uuid: "5c629c8bc36e4d4893d30f9a7c91f869"
format_stamp: "Formatted at 2026-08-10 23:25:27 on dist-test-slave-rmhg"
server_key: "2dc86f7fafb68c62f18c2ae0316672ed"
server_key_iv: "62a7811f985cb76ad12255c91d6edd3a"
server_key_version: "encryptionkey@0"
I20260810 23:25:27.103554 19253 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:25:27.107429 19276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:27.109009 19253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:25:27.109314 19253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-1/wal
uuid: "5c629c8bc36e4d4893d30f9a7c91f869"
format_stamp: "Formatted at 2026-08-10 23:25:27 on dist-test-slave-rmhg"
server_key: "2dc86f7fafb68c62f18c2ae0316672ed"
server_key_iv: "62a7811f985cb76ad12255c91d6edd3a"
server_key_version: "encryptionkey@0"
I20260810 23:25:27.109581 19253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:27.140919 19253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:27.145922 19253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:27.146234 19253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:27.147320 19253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:25:27.148833 19253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:25:27.148921 19253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:27.149052 19253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:25:27.149117 19253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:27.180497 19253 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.66:43857
I20260810 23:25:27.180541 19445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.66:43857 every 8 connection(s)
I20260810 23:25:27.181867 19253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
I20260810 23:25:27.185333 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 19253
I20260810 23:25:27.185520 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance
I20260810 23:25:27.186134 15037 external_mini_cluster.cc:1428] Setting key 07e24555859ca648dba600ca1b4c58c7
Aug 10 23:25:27 dist-test-slave-rmhg krb5kdc[18881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404327, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:27.204602 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:27.184537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:52271 (local address 127.14.175.126:36599)
0810 23:25:27.184991 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:27.184998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:27.186159 (+ 1161us) server_negotiation.cc:407] Connection header received
0810 23:25:27.188329 (+ 2170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:27.188333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:27.188398 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:27.188485 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:27.190676 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:27.191464 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:27.193754 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:27.194196 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:27.194292 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:27.198114 (+ 3822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:27.198120 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:27.198126 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:27.198169 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:27.201357 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:27.202127 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:27.202133 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:27.202136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:27.202289 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:27.202713 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:27.202718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:27.202722 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:27.203070 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:27.203209 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:27.203928 (+ 719us) server_negotiation.cc:299] Negotiation successful
0810 23:25:27.204178 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":220,"threads_started":1}
I20260810 23:25:27.205549 19452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:27.185069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:36599 (local address 127.14.175.66:52271)
0810 23:25:27.185953 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:27.186007 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:27.188049 (+ 2042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:27.188791 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:27.188805 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:27.189605 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:27.190468 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:27.190496 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:27.191676 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:27.191682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:27.193603 (+ 1921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:27.193620 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:27.194419 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:27.194426 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:27.194711 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:27.195426 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:27.195456 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:27.197820 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:27.201513 (+ 3693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:27.201522 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:27.201536 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:27.201984 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:27.202382 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:27.202385 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:27.202387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:27.202582 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:27.203187 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:27.203193 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:27.203545 (+ 352us) client_negotiation.cc:769] Sending connection context
0810 23:25:27.203778 (+ 233us) client_negotiation.cc:240] Negotiation successful
0810 23:25:27.204215 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":337,"threads_started":1}
I20260810 23:25:27.207038 19446 heartbeater.cc:347] Connected to a master server at 127.14.175.126:36599
I20260810 23:25:27.207336 19446 heartbeater.cc:464] Registering TS with master...
I20260810 23:25:27.208071 19446 heartbeater.cc:511] Master 127.14.175.126:36599 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.175.67@KRBTEST.COM; defaulting to no policy
I20260810 23:25:27.209667 18959 ts_manager.cc:194] Registered new tserver with Master: 5c629c8bc36e4d4893d30f9a7c91f869 (127.14.175.66:43857)
I20260810 23:25:27.210523 18959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.66@KRBTEST.COM'} at 127.14.175.66:52271
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
I20260810 23:25:27.236809 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.67:0
--local_ip_for_outbound_sockets=127.14.175.67
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:36599
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:37047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:27.574997 19455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:27.575321 19455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:27.575456 19455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:27.583076 19455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:27.583218 19455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:27.583338 19455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.67
I20260810 23:25:27.593713 19455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:37047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:36599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.67
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:27.595052 19455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:27.596405 19455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:27.610411 19461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:27.610450 19463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:27.610515 19469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:27.611407 19455 server_base.cc:1034] running on GCE node
I20260810 23:25:27.612102 19455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:27.613137 19455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:27.614409 19455 hybrid_clock.cc:648] HybridClock initialized: now 1786404327614359 us; error 43 us; skew 500 ppm
I20260810 23:25:27.614785 19455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:27 dist-test-slave-rmhg krb5kdc[18881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404327, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:27.618220 19455 init.cc:377] Logged in from keytab as kudu/127.14.175.67@KRBTEST.COM (short username kudu)
I20260810 23:25:27.619866 19455 webserver.cc:460] Webserver started at http://127.14.175.67:35137/ using document root <none> and password file <none>
I20260810 23:25:27.620651 19455 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:27.620791 19455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:27.621137 19455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:27.624372 19455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-2/data/instance:
uuid: "e13819bac2f24194a1b95188bde827c2"
format_stamp: "Formatted at 2026-08-10 23:25:27 on dist-test-slave-rmhg"
server_key: "a10185bfbaa10f1395d73767b8430e35"
server_key_iv: "01a5104f91c2014fc257f036144c6c8d"
server_key_version: "encryptionkey@0"
I20260810 23:25:27.625303 19455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance:
uuid: "e13819bac2f24194a1b95188bde827c2"
format_stamp: "Formatted at 2026-08-10 23:25:27 on dist-test-slave-rmhg"
server_key: "a10185bfbaa10f1395d73767b8430e35"
server_key_iv: "01a5104f91c2014fc257f036144c6c8d"
server_key_version: "encryptionkey@0"
I20260810 23:25:27.631335 19455 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 23:25:27.635190 19478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:27.636674 19455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:25:27.636862 19455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-2/wal
uuid: "e13819bac2f24194a1b95188bde827c2"
format_stamp: "Formatted at 2026-08-10 23:25:27 on dist-test-slave-rmhg"
server_key: "a10185bfbaa10f1395d73767b8430e35"
server_key_iv: "01a5104f91c2014fc257f036144c6c8d"
server_key_version: "encryptionkey@0"
I20260810 23:25:27.637090 19455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:27.660301 19455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:27.664994 19455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:27.665329 19455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:27.666360 19455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:25:27.667770 19455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:25:27.667869 19455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:27.667992 19455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:25:27.668040 19455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:27.685060 19243 heartbeater.cc:503] Master 127.14.175.126:36599 was elected leader, sending a full tablet report...
I20260810 23:25:27.699038 19455 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.67:46091
I20260810 23:25:27.699086 19642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.67:46091 every 8 connection(s)
I20260810 23:25:27.700407 19455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
I20260810 23:25:27.708895 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 19455
I20260810 23:25:27.709061 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance
I20260810 23:25:27.709551 15037 external_mini_cluster.cc:1428] Setting key 8b2baf95908b2539bffd1d4d9269241f
Aug 10 23:25:27 dist-test-slave-rmhg krb5kdc[18881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404327, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:27.721486 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:27.703203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:50135 (local address 127.14.175.126:36599)
0810 23:25:27.703418 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:27.703423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:27.704687 (+ 1264us) server_negotiation.cc:407] Connection header received
0810 23:25:27.706642 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:27.706648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:27.706757 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:27.706846 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:27.708555 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:27.709411 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:27.711262 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:27.711785 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:27.711896 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:27.715624 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:27.715630 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:27.715634 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:27.715671 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:27.718316 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:27.719134 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:27.719138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:27.719140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:27.719281 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:27.719663 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:27.719666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:27.719668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:27.719962 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:27.720096 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:27.720960 (+ 864us) server_negotiation.cc:299] Negotiation successful
0810 23:25:27.721239 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 23:25:27.722559 19647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:27.703642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:36599 (local address 127.14.175.67:50135)
0810 23:25:27.704450 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:27.704529 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:27.706325 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:27.707128 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:27.707138 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:27.707726 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:27.708350 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:27.708369 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:27.709571 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:27.709575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:27.711117 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:27.711130 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:27.712029 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:27.712038 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:27.712272 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:27.712989 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:27.713014 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:27.715336 (+ 2322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:27.718481 (+ 3145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:27.718490 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:27.718508 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:27.718995 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:27.719375 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:27.719378 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:27.719380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:27.719578 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:27.720126 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:27.720133 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:27.720530 (+ 397us) client_negotiation.cc:769] Sending connection context
0810 23:25:27.720800 (+ 270us) client_negotiation.cc:240] Negotiation successful
0810 23:25:27.721309 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":324,"threads_started":1}
I20260810 23:25:27.724366 19643 heartbeater.cc:347] Connected to a master server at 127.14.175.126:36599
I20260810 23:25:27.724767 19643 heartbeater.cc:464] Registering TS with master...
I20260810 23:25:27.725616 19643 heartbeater.cc:511] Master 127.14.175.126:36599 requested a full tablet report, sending...
I20260810 23:25:27.727141 18957 ts_manager.cc:194] Registered new tserver with Master: e13819bac2f24194a1b95188bde827c2 (127.14.175.67:46091)
I20260810 23:25:27.728025 18957 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.67@KRBTEST.COM'} at 127.14.175.67:50135
I20260810 23:25:27.737988 15037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 23:25:27.775853 19448 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:25:27.772798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53754 (local address 127.14.175.126:36599)
0810 23:25:27.773042 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:27.773049 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:27.773444 (+ 395us) server_negotiation.cc:407] Connection header received
0810 23:25:27.773667 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:27.773671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:27.773768 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:27.773860 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:27.775520 (+ 1660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53754: BlockingRecv error: recv got EOF from 127.0.0.1:53754 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 23:25:27.779327 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 19051
W20260810 23:25:27.782547 15037 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 19051: Not authorized: Client connection negotiation failed: client connection to 127.14.175.65:45851: server requires authentication, but client does not have Kerberos credentials available
I20260810 23:25:27.782799 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 19051
I20260810 23:25:27.805991 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 19253
W20260810 23:25:27.808933 15037 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 19253: Not authorized: Client connection negotiation failed: client connection to 127.14.175.66:43857: server requires authentication, but client does not have Kerberos credentials available
I20260810 23:25:27.809113 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 19253
W20260810 23:25:27.809242 19662 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:25:27.806985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41194 (local address 127.14.175.66:43857)
0810 23:25:27.807558 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:27.807565 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:27.807594 (+ 29us) server_negotiation.cc:407] Connection header received
0810 23:25:27.807657 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:27.807663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:27.807898 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:27.808041 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:27.808835 (+ 794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41194: BlockingRecv error: recv got EOF from 127.0.0.1:41194 (error 108)
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":302,"threads_started":1}
I20260810 23:25:27.831489 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 19455
W20260810 23:25:27.834558 15037 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 19455: Not authorized: Client connection negotiation failed: client connection to 127.14.175.67:46091: server requires authentication, but client does not have Kerberos credentials available
I20260810 23:25:27.834756 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 19455
I20260810 23:25:27.874236 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 18901
I20260810 23:25:27.875566 19003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:25:27.980968 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 18901
2026-08-10T23:25:28Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3884 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:25:28 dist-test-slave-rmhg krb5kdc[19670](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:25:28 dist-test-slave-rmhg krb5kdc[19670](info): set up 2 sockets
Aug 10 23:25:28 dist-test-slave-rmhg krb5kdc[19670](info): commencing operation
krb5kdc: starting...
W20260810 23:25:29.204937 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.172s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:25:29 dist-test-slave-rmhg krb5kdc[19670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:25:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:25:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
I20260810 23:25:29.441529 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:40757
--webserver_interface=127.14.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:43623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:40757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:29.793572 19693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:29.793920 19693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:29.794059 19693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:29.801613 19693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:25:29.801760 19693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:29.801817 19693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:29.801900 19693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:25:29.801966 19693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:25:29.812381 19693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:43623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:40757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:40757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:29.813858 19693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:29.815289 19693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:29.827859 19702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:29.827945 19703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:29.827908 19705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:29.828802 19693 server_base.cc:1034] running on GCE node
I20260810 23:25:29.829712 19693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:29.831229 19693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:29.832460 19693 hybrid_clock.cc:648] HybridClock initialized: now 1786404329832401 us; error 50 us; skew 500 ppm
I20260810 23:25:29.832909 19693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:29 dist-test-slave-rmhg krb5kdc[19670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404329, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:29.836225 19693 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:25:29.837947 19693 webserver.cc:460] Webserver started at http://127.14.175.126:44821/ using document root <none> and password file <none>
I20260810 23:25:29.838649 19693 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:29.838770 19693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:29.839049 19693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:29.842298 19693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "b4ca358e26a944088fab61881f061985"
format_stamp: "Formatted at 2026-08-10 23:25:29 on dist-test-slave-rmhg"
server_key: "641340a6f760fff9528e3bc1c24c7600"
server_key_iv: "9f8e146c2ff7caf62d163fd89c31e8ca"
server_key_version: "encryptionkey@0"
I20260810 23:25:29.843128 19693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "b4ca358e26a944088fab61881f061985"
format_stamp: "Formatted at 2026-08-10 23:25:29 on dist-test-slave-rmhg"
server_key: "641340a6f760fff9528e3bc1c24c7600"
server_key_iv: "9f8e146c2ff7caf62d163fd89c31e8ca"
server_key_version: "encryptionkey@0"
I20260810 23:25:29.848927 19693 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:25:29.852946 19718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:29.854535 19693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:25:29.854743 19693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "b4ca358e26a944088fab61881f061985"
format_stamp: "Formatted at 2026-08-10 23:25:29 on dist-test-slave-rmhg"
server_key: "641340a6f760fff9528e3bc1c24c7600"
server_key_iv: "9f8e146c2ff7caf62d163fd89c31e8ca"
server_key_version: "encryptionkey@0"
I20260810 23:25:29.854964 19693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:29.892051 19693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:29.896586 19693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:29.896899 19693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:29.915604 19693 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:40757
I20260810 23:25:29.915642 19792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:40757 every 8 connection(s)
I20260810 23:25:29.917111 19693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:25:29.921372 19794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:29.925061 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 19693
I20260810 23:25:29.925370 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:25:29.926177 15037 external_mini_cluster.cc:1428] Setting key 4e396a8cdd4ad5d378a411ebe8665c2a
I20260810 23:25:29.935386 19794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4ca358e26a944088fab61881f061985: Bootstrap starting.
I20260810 23:25:29.939502 19794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4ca358e26a944088fab61881f061985: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:29.940933 19794 log.cc:824] T 00000000000000000000000000000000 P b4ca358e26a944088fab61881f061985: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:25:29 dist-test-slave-rmhg krb5kdc[19670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:29.944209 19794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4ca358e26a944088fab61881f061985: No bootstrap required, opened a new log
I20260810 23:25:29.950579 19797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:29.928317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37156 (local address 127.14.175.126:40757)
0810 23:25:29.929272 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:29.929286 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:29.929332 (+ 46us) server_negotiation.cc:407] Connection header received
0810 23:25:29.931099 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:29.931127 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:29.931604 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:29.932039 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:29.933949 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:29.935364 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:29.937200 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:29.937952 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:29.938127 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:29.943127 (+ 5000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:29.943136 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:29.943154 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:29.943193 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:29.946310 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:29.947024 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:29.947031 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:29.947038 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:29.947224 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:29.947613 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:29.947617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:29.947619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:29.948212 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:29.948539 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:29.949003 (+ 464us) server_negotiation.cc:299] Negotiation successful
0810 23:25:29.949499 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":443,"threads_started":1}
I20260810 23:25:29.952409 19794 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4ca358e26a944088fab61881f061985 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ca358e26a944088fab61881f061985" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 40757 } }
I20260810 23:25:29.952852 19794 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4ca358e26a944088fab61881f061985 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:29.952962 19794 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4ca358e26a944088fab61881f061985 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4ca358e26a944088fab61881f061985, State: Initialized, Role: FOLLOWER
I20260810 23:25:29.953642 19794 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4ca358e26a944088fab61881f061985 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ca358e26a944088fab61881f061985" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 40757 } }
I20260810 23:25:29.953917 19794 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4ca358e26a944088fab61881f061985 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:25:29.954063 19794 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4ca358e26a944088fab61881f061985 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:25:29.954260 19794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4ca358e26a944088fab61881f061985 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:29.954736 15037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:25:29.956524 19794 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4ca358e26a944088fab61881f061985 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ca358e26a944088fab61881f061985" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 40757 } }
I20260810 23:25:29.956993 19794 leader_election.cc:302] T 00000000000000000000000000000000 P b4ca358e26a944088fab61881f061985 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4ca358e26a944088fab61881f061985; no voters:
I20260810 23:25:29.957628 19794 leader_election.cc:288] T 00000000000000000000000000000000 P b4ca358e26a944088fab61881f061985 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:25:29.957785 19799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4ca358e26a944088fab61881f061985 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:25:29.958868 19799 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4ca358e26a944088fab61881f061985 [term 1 LEADER]: Becoming Leader. State: Replica: b4ca358e26a944088fab61881f061985, State: Running, Role: LEADER
I20260810 23:25:29.959493 19799 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4ca358e26a944088fab61881f061985 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ca358e26a944088fab61881f061985" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 40757 } }
I20260810 23:25:29.959870 19794 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4ca358e26a944088fab61881f061985 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:25:29.970268 19801 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4ca358e26a944088fab61881f061985 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4ca358e26a944088fab61881f061985. Latest consensus state: current_term: 1 leader_uuid: "b4ca358e26a944088fab61881f061985" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ca358e26a944088fab61881f061985" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 40757 } } }
I20260810 23:25:29.970763 19801 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4ca358e26a944088fab61881f061985 [sys.catalog]: This master's current role is: LEADER
I20260810 23:25:29.970963 19800 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4ca358e26a944088fab61881f061985 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4ca358e26a944088fab61881f061985" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ca358e26a944088fab61881f061985" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 40757 } } }
I20260810 23:25:29.971316 19800 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4ca358e26a944088fab61881f061985 [sys.catalog]: This master's current role is: LEADER
I20260810 23:25:29.972078 19810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:25:29.979077 19810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:25:29.998582 19810 catalog_manager.cc:1347] Generated new cluster ID: a0b5f23808e949fe806bdd4d03e54509
I20260810 23:25:29.998729 19810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:25:30.014343 19810 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:25:30.016535 19810 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:25:30.025701 19810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4ca358e26a944088fab61881f061985: Generated new TSK 0
I20260810 23:25:30.026947 19810 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:25:30.039212 19797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:30.031887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37164 (local address 127.14.175.126:40757)
0810 23:25:30.032060 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:30.032065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:30.032152 (+ 87us) server_negotiation.cc:407] Connection header received
0810 23:25:30.032341 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:30.032346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:30.032416 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:30.032507 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:30.033639 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:30.034394 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:30.035720 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:30.036159 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:30.036273 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:30.037495 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:30.037501 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:30.037504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:25:30.038412 (+ 908us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:30.038480 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:25:30.038510 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:30.038636 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:30.038767 (+ 131us) server_negotiation.cc:299] Negotiation successful
0810 23:25:30.038878 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 23:25:30.047590 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 19693
I20260810 23:25:30.048771 19787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:25:30.144364 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 19693
2026-08-10T23:25:30Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2158 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:25:30 dist-test-slave-rmhg krb5kdc[19836](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:25:30 dist-test-slave-rmhg krb5kdc[19836](info): set up 2 sockets
Aug 10 23:25:30 dist-test-slave-rmhg krb5kdc[19836](info): commencing operation
krb5kdc: starting...
W20260810 23:25:31.364323 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.173s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:25:31 dist-test-slave-rmhg krb5kdc[19836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:25:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:25:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
I20260810 23:25:31.593099 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:45381
--webserver_interface=127.14.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:42823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:45381
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:31.946967 19855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:31.947285 19855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:31.947412 19855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:31.954910 19855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:25:31.955037 19855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:31.955097 19855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:31.955183 19855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:25:31.955250 19855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:25:31.965657 19855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:42823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:45381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:45381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:31.967100 19855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:31.968595 19855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:31.981222 19864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:31.981366 19863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:31.981390 19868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:31.982211 19855 server_base.cc:1034] running on GCE node
I20260810 23:25:31.983065 19855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:31.984632 19855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:31.985874 19855 hybrid_clock.cc:648] HybridClock initialized: now 1786404331985824 us; error 35 us; skew 500 ppm
I20260810 23:25:31.986261 19855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:31 dist-test-slave-rmhg krb5kdc[19836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404331, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:31.994360 19855 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:25:31.996228 19855 webserver.cc:460] Webserver started at http://127.14.175.126:46015/ using document root <none> and password file <none>
I20260810 23:25:31.996979 19855 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:31.997112 19855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:31.997467 19855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:32.000732 19855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "a29c9c4336284046b35f56e245cb6674"
format_stamp: "Formatted at 2026-08-10 23:25:31 on dist-test-slave-rmhg"
server_key: "99e8c9e7b81550cd1234dc070d2d7da8"
server_key_iv: "b9a4a03bce3a28f5fc922cade0a43496"
server_key_version: "encryptionkey@0"
I20260810 23:25:32.001652 19855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "a29c9c4336284046b35f56e245cb6674"
format_stamp: "Formatted at 2026-08-10 23:25:31 on dist-test-slave-rmhg"
server_key: "99e8c9e7b81550cd1234dc070d2d7da8"
server_key_iv: "b9a4a03bce3a28f5fc922cade0a43496"
server_key_version: "encryptionkey@0"
I20260810 23:25:32.007651 19855 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 23:25:32.011390 19880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:32.012867 19855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 23:25:32.013060 19855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "a29c9c4336284046b35f56e245cb6674"
format_stamp: "Formatted at 2026-08-10 23:25:31 on dist-test-slave-rmhg"
server_key: "99e8c9e7b81550cd1234dc070d2d7da8"
server_key_iv: "b9a4a03bce3a28f5fc922cade0a43496"
server_key_version: "encryptionkey@0"
I20260810 23:25:32.013299 19855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:32.037693 19855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:32.042086 19855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:32.042402 19855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:32.061038 19855 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:45381
I20260810 23:25:32.061056 19954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:45381 every 8 connection(s)
I20260810 23:25:32.062552 19855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:25:32.066537 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 19855
I20260810 23:25:32.066747 19955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:32.066807 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:25:32.067605 15037 external_mini_cluster.cc:1428] Setting key b3c2e3cd923f7ae7381ef62d27075782
I20260810 23:25:32.081789 19955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a29c9c4336284046b35f56e245cb6674: Bootstrap starting.
Aug 10 23:25:32 dist-test-slave-rmhg krb5kdc[19836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:32.086210 19955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a29c9c4336284046b35f56e245cb6674: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:32.087447 19955 log.cc:824] T 00000000000000000000000000000000 P a29c9c4336284046b35f56e245cb6674: Log is configured to *not* fsync() on all Append() calls
I20260810 23:25:32.091384 19955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a29c9c4336284046b35f56e245cb6674: No bootstrap required, opened a new log
I20260810 23:25:32.095178 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:32.069858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58332 (local address 127.14.175.126:45381)
0810 23:25:32.070714 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:32.070727 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:32.070773 (+ 46us) server_negotiation.cc:407] Connection header received
0810 23:25:32.073272 (+ 2499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:32.073301 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:32.073799 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:32.074172 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:32.076049 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:32.077481 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:32.079414 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:32.080233 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:32.080390 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:32.085762 (+ 5372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:32.085774 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:32.085793 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:32.085847 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:32.089510 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:32.090459 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:32.090469 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:32.090479 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:32.090727 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:32.091215 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:32.091220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:32.091223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:32.092045 (+ 822us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:32.092396 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:32.092923 (+ 527us) server_negotiation.cc:299] Negotiation successful
0810 23:25:32.093510 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":397,"threads_started":1}
I20260810 23:25:32.100661 19955 raft_consensus.cc:348] T 00000000000000000000000000000000 P a29c9c4336284046b35f56e245cb6674 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29c9c4336284046b35f56e245cb6674" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 45381 } }
I20260810 23:25:32.101081 19955 raft_consensus.cc:374] T 00000000000000000000000000000000 P a29c9c4336284046b35f56e245cb6674 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:32.101223 19955 raft_consensus.cc:729] T 00000000000000000000000000000000 P a29c9c4336284046b35f56e245cb6674 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29c9c4336284046b35f56e245cb6674, State: Initialized, Role: FOLLOWER
I20260810 23:25:32.101838 19955 consensus_queue.cc:260] T 00000000000000000000000000000000 P a29c9c4336284046b35f56e245cb6674 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29c9c4336284046b35f56e245cb6674" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 45381 } }
I20260810 23:25:32.102061 19955 raft_consensus.cc:388] T 00000000000000000000000000000000 P a29c9c4336284046b35f56e245cb6674 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:25:32.102208 19955 raft_consensus.cc:482] T 00000000000000000000000000000000 P a29c9c4336284046b35f56e245cb6674 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:25:32.102376 19955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a29c9c4336284046b35f56e245cb6674 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:32.104638 19955 raft_consensus.cc:504] T 00000000000000000000000000000000 P a29c9c4336284046b35f56e245cb6674 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29c9c4336284046b35f56e245cb6674" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 45381 } }
I20260810 23:25:32.105230 19955 leader_election.cc:302] T 00000000000000000000000000000000 P a29c9c4336284046b35f56e245cb6674 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a29c9c4336284046b35f56e245cb6674; no voters:
I20260810 23:25:32.105839 19955 leader_election.cc:288] T 00000000000000000000000000000000 P a29c9c4336284046b35f56e245cb6674 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:25:32.105984 19962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a29c9c4336284046b35f56e245cb6674 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:25:32.106663 19962 raft_consensus.cc:686] T 00000000000000000000000000000000 P a29c9c4336284046b35f56e245cb6674 [term 1 LEADER]: Becoming Leader. State: Replica: a29c9c4336284046b35f56e245cb6674, State: Running, Role: LEADER
I20260810 23:25:32.107229 19962 consensus_queue.cc:237] T 00000000000000000000000000000000 P a29c9c4336284046b35f56e245cb6674 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29c9c4336284046b35f56e245cb6674" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 45381 } }
I20260810 23:25:32.108026 19955 sys_catalog.cc:564] T 00000000000000000000000000000000 P a29c9c4336284046b35f56e245cb6674 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:25:32.110546 19963 sys_catalog.cc:455] T 00000000000000000000000000000000 P a29c9c4336284046b35f56e245cb6674 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a29c9c4336284046b35f56e245cb6674" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29c9c4336284046b35f56e245cb6674" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 45381 } } }
I20260810 23:25:32.110610 19964 sys_catalog.cc:455] T 00000000000000000000000000000000 P a29c9c4336284046b35f56e245cb6674 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a29c9c4336284046b35f56e245cb6674. Latest consensus state: current_term: 1 leader_uuid: "a29c9c4336284046b35f56e245cb6674" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29c9c4336284046b35f56e245cb6674" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 45381 } } }
I20260810 23:25:32.110980 19963 sys_catalog.cc:458] T 00000000000000000000000000000000 P a29c9c4336284046b35f56e245cb6674 [sys.catalog]: This master's current role is: LEADER
I20260810 23:25:32.111023 19964 sys_catalog.cc:458] T 00000000000000000000000000000000 P a29c9c4336284046b35f56e245cb6674 [sys.catalog]: This master's current role is: LEADER
I20260810 23:25:32.113591 19972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:25:32.119014 19972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:25:32.129484 19972 catalog_manager.cc:1347] Generated new cluster ID: c183cca3128741e9b3900c1414b92246
I20260810 23:25:32.129638 19972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:25:32.137228 19972 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:25:32.138814 19972 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:25:32.148659 19972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a29c9c4336284046b35f56e245cb6674: Generated new TSK 0
I20260810 23:25:32.149966 19972 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
I20260810 23:25:32.206686 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.65:0
--local_ip_for_outbound_sockets=127.14.175.65
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:45381
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:42823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:32.560441 19987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:32.560751 19987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:32.560886 19987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:32.568535 19987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:32.568662 19987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:32.568814 19987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.65
I20260810 23:25:32.579067 19987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:42823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:45381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.65
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:32.580694 19987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:32.582162 19987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:32.595703 19996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:32.595777 20000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:32.596134 19995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:32.596989 19987 server_base.cc:1034] running on GCE node
I20260810 23:25:32.597775 19987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:32.598868 19987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:32.600126 19987 hybrid_clock.cc:648] HybridClock initialized: now 1786404332600071 us; error 49 us; skew 500 ppm
I20260810 23:25:32.600471 19987 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:32 dist-test-slave-rmhg krb5kdc[19836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404332, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:32.603859 19987 init.cc:377] Logged in from keytab as kudu/127.14.175.65@KRBTEST.COM (short username kudu)
I20260810 23:25:32.605540 19987 webserver.cc:460] Webserver started at http://127.14.175.65:37537/ using document root <none> and password file <none>
I20260810 23:25:32.606254 19987 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:32.606365 19987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:32.606643 19987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:32.609936 19987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-0/data/instance:
uuid: "6fced4a9788e459c8b39f6e932a27b2d"
format_stamp: "Formatted at 2026-08-10 23:25:32 on dist-test-slave-rmhg"
server_key: "8c0b2113e33839c61e113ac1359c43b3"
server_key_iv: "5cebdbba5bbf720c7735ebde495ae872"
server_key_version: "encryptionkey@0"
I20260810 23:25:32.610743 19987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance:
uuid: "6fced4a9788e459c8b39f6e932a27b2d"
format_stamp: "Formatted at 2026-08-10 23:25:32 on dist-test-slave-rmhg"
server_key: "8c0b2113e33839c61e113ac1359c43b3"
server_key_iv: "5cebdbba5bbf720c7735ebde495ae872"
server_key_version: "encryptionkey@0"
I20260810 23:25:32.616556 19987 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 23:25:32.620594 20013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:32.622381 19987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:25:32.622556 19987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-0/wal
uuid: "6fced4a9788e459c8b39f6e932a27b2d"
format_stamp: "Formatted at 2026-08-10 23:25:32 on dist-test-slave-rmhg"
server_key: "8c0b2113e33839c61e113ac1359c43b3"
server_key_iv: "5cebdbba5bbf720c7735ebde495ae872"
server_key_version: "encryptionkey@0"
I20260810 23:25:32.622773 19987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:32.649636 19987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:32.654227 19987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:32.654502 19987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:32.655478 19987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:25:32.656919 19987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:25:32.656998 19987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:32.657094 19987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:25:32.657138 19987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:32.689055 19987 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.65:42687
I20260810 23:25:32.689100 20191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.65:42687 every 8 connection(s)
I20260810 23:25:32.690446 19987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
I20260810 23:25:32.690855 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 19987
I20260810 23:25:32.691072 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance
I20260810 23:25:32.691882 15037 external_mini_cluster.cc:1428] Setting key a6210b39c91213ec343b10eb1fb66999
Aug 10 23:25:32 dist-test-slave-rmhg krb5kdc[19836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404332, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.175.66@KRBTEST.COM; defaulting to no policy
I20260810 23:25:32.713119 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:32.693356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:49739 (local address 127.14.175.126:45381)
0810 23:25:32.693837 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:32.693844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:32.695094 (+ 1250us) server_negotiation.cc:407] Connection header received
0810 23:25:32.697640 (+ 2546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:32.697645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:32.697732 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:32.697831 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:32.699993 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:32.700749 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:32.702831 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:32.703255 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:32.703380 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:32.707046 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:32.707052 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:32.707055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:32.707090 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:32.709917 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:32.710718 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:32.710724 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:32.710727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:32.710857 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:32.711301 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:32.711305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:32.711308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:32.711579 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:32.711718 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:32.712506 (+ 788us) server_negotiation.cc:299] Negotiation successful
0810 23:25:32.712755 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":230,"threads_started":1}
I20260810 23:25:32.714330 20196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:32.694009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45381 (local address 127.14.175.65:49739)
0810 23:25:32.694891 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:32.694959 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:32.697312 (+ 2353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:32.698131 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:32.698144 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:32.698929 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:32.699798 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:32.699826 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:32.700936 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:32.700941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:32.702635 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:32.702655 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:32.703507 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:32.703514 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:32.703767 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:32.704451 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:32.704490 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:32.706792 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:32.710073 (+ 3281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:32.710084 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:32.710103 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:32.710576 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:32.710966 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:32.710971 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:32.710974 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:32.711177 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:32.711718 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:32.711727 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:32.712099 (+ 372us) client_negotiation.cc:769] Sending connection context
0810 23:25:32.712354 (+ 255us) client_negotiation.cc:240] Negotiation successful
0810 23:25:32.713015 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":327,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.66@KRBTEST.COM" created.
I20260810 23:25:32.716089 20192 heartbeater.cc:347] Connected to a master server at 127.14.175.126:45381
I20260810 23:25:32.716475 20192 heartbeater.cc:464] Registering TS with master...
I20260810 23:25:32.717494 20192 heartbeater.cc:511] Master 127.14.175.126:45381 requested a full tablet report, sending...
I20260810 23:25:32.720044 19907 ts_manager.cc:194] Registered new tserver with Master: 6fced4a9788e459c8b39f6e932a27b2d (127.14.175.65:42687)
I20260810 23:25:32.721616 19907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.65@KRBTEST.COM'} at 127.14.175.65:49739
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
I20260810 23:25:32.742156 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.66:0
--local_ip_for_outbound_sockets=127.14.175.66
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:45381
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:42823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:33.097394 20202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:33.097712 20202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:33.097810 20202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:33.105365 20202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:33.105495 20202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:33.105620 20202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.66
I20260810 23:25:33.116196 20202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:42823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:45381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.66
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:33.117657 20202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:33.119069 20202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:33.134143 20208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:33.134112 20211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:33.134600 20207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:33.135080 20202 server_base.cc:1034] running on GCE node
I20260810 23:25:33.135756 20202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:33.136878 20202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:33.138135 20202 hybrid_clock.cc:648] HybridClock initialized: now 1786404333138053 us; error 60 us; skew 500 ppm
I20260810 23:25:33.138500 20202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:33 dist-test-slave-rmhg krb5kdc[19836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404333, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:33.141963 20202 init.cc:377] Logged in from keytab as kudu/127.14.175.66@KRBTEST.COM (short username kudu)
I20260810 23:25:33.143683 20202 webserver.cc:460] Webserver started at http://127.14.175.66:41681/ using document root <none> and password file <none>
I20260810 23:25:33.144426 20202 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:33.144586 20202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:33.144898 20202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:33.148123 20202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-1/data/instance:
uuid: "e97bae6c0c3a4180b0d429cedfe86f17"
format_stamp: "Formatted at 2026-08-10 23:25:33 on dist-test-slave-rmhg"
server_key: "68934cf386ecb888980f4e27657eb8e2"
server_key_iv: "90e50d48422eb71d7e6b653b57611a76"
server_key_version: "encryptionkey@0"
I20260810 23:25:33.148940 20202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance:
uuid: "e97bae6c0c3a4180b0d429cedfe86f17"
format_stamp: "Formatted at 2026-08-10 23:25:33 on dist-test-slave-rmhg"
server_key: "68934cf386ecb888980f4e27657eb8e2"
server_key_iv: "90e50d48422eb71d7e6b653b57611a76"
server_key_version: "encryptionkey@0"
I20260810 23:25:33.154738 20202 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 23:25:33.158768 20220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:33.160459 20202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260810 23:25:33.160671 20202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-1/wal
uuid: "e97bae6c0c3a4180b0d429cedfe86f17"
format_stamp: "Formatted at 2026-08-10 23:25:33 on dist-test-slave-rmhg"
server_key: "68934cf386ecb888980f4e27657eb8e2"
server_key_iv: "90e50d48422eb71d7e6b653b57611a76"
server_key_version: "encryptionkey@0"
I20260810 23:25:33.160887 20202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:33.176676 20202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:33.181695 20202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:33.182001 20202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:33.183032 20202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:25:33.184566 20202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:25:33.184649 20202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:33.184789 20202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:25:33.184846 20202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:33.217447 20202 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.66:34131
I20260810 23:25:33.217504 20384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.66:34131 every 8 connection(s)
I20260810 23:25:33.218861 20202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
I20260810 23:25:33.228354 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 20202
I20260810 23:25:33.228539 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance
I20260810 23:25:33.229002 15037 external_mini_cluster.cc:1428] Setting key 42b966d9acc692a2b225640d4f5492c8
Aug 10 23:25:33 dist-test-slave-rmhg krb5kdc[19836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404333, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:33.241410 20387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:33.221630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:50421 (local address 127.14.175.126:45381)
0810 23:25:33.222152 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:33.222157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:33.223149 (+ 992us) server_negotiation.cc:407] Connection header received
0810 23:25:33.225039 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:33.225045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:33.225140 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:33.225273 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:33.227013 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:33.227754 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:33.229742 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:33.230240 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:33.230344 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:33.234314 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:33.234321 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:33.234326 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:33.234370 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:33.237636 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:33.238573 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:33.238580 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:33.238583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:33.238730 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:33.239355 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:33.239360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:33.239363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:33.239725 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:33.239868 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:33.240857 (+ 989us) server_negotiation.cc:299] Negotiation successful
0810 23:25:33.241065 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":307,"threads_started":1}
I20260810 23:25:33.243080 20388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:33.222155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45381 (local address 127.14.175.66:50421)
0810 23:25:33.222943 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:33.223012 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:33.224729 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:33.225550 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:33.225560 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:33.226171 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:33.226804 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:33.226824 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:33.227931 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:33.227935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:33.229558 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:33.229578 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:33.230493 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:33.230502 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:33.230834 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:33.231652 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:33.231677 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:33.234027 (+ 2350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:33.237835 (+ 3808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:33.237845 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:33.237862 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:33.238415 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:33.238844 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:33.238849 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:33.238852 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:33.239173 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:33.239873 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:33.239883 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:33.240396 (+ 513us) client_negotiation.cc:769] Sending connection context
0810 23:25:33.240744 (+ 348us) client_negotiation.cc:240] Negotiation successful
0810 23:25:33.241403 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":298,"threads_started":1}
I20260810 23:25:33.244822 20385 heartbeater.cc:347] Connected to a master server at 127.14.175.126:45381
I20260810 23:25:33.245231 20385 heartbeater.cc:464] Registering TS with master...
I20260810 23:25:33.246237 20385 heartbeater.cc:511] Master 127.14.175.126:45381 requested a full tablet report, sending...
I20260810 23:25:33.248167 19907 ts_manager.cc:194] Registered new tserver with Master: e97bae6c0c3a4180b0d429cedfe86f17 (127.14.175.66:34131)
I20260810 23:25:33.249385 19907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.66@KRBTEST.COM'} at 127.14.175.66:50421
WARNING: no policy specified for kudu/127.14.175.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
I20260810 23:25:33.284474 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.67:0
--local_ip_for_outbound_sockets=127.14.175.67
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:45381
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:42823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:33.626194 20393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:33.626523 20393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:33.626664 20393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:33.634187 20393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:33.634321 20393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:33.634483 20393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.67
I20260810 23:25:33.644636 20393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:42823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:45381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.67
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:33.646085 20393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:33.647526 20393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:33.661592 20402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:33.661686 20405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:33.661561 20401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:33.662305 20393 server_base.cc:1034] running on GCE node
I20260810 23:25:33.662997 20393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:33.664023 20393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:33.665246 20393 hybrid_clock.cc:648] HybridClock initialized: now 1786404333665197 us; error 39 us; skew 500 ppm
I20260810 23:25:33.665576 20393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:33 dist-test-slave-rmhg krb5kdc[19836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404333, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:33.668792 20393 init.cc:377] Logged in from keytab as kudu/127.14.175.67@KRBTEST.COM (short username kudu)
I20260810 23:25:33.670398 20393 webserver.cc:460] Webserver started at http://127.14.175.67:41717/ using document root <none> and password file <none>
I20260810 23:25:33.671149 20393 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:33.671278 20393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:33.671594 20393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:33.674726 20393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-2/data/instance:
uuid: "d116956c15d34375898f83fcebccd32b"
format_stamp: "Formatted at 2026-08-10 23:25:33 on dist-test-slave-rmhg"
server_key: "adac8986cbf698e160f4c5508f6c6ea6"
server_key_iv: "44941b3a502b968168445fabe5730f83"
server_key_version: "encryptionkey@0"
I20260810 23:25:33.675537 20393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance:
uuid: "d116956c15d34375898f83fcebccd32b"
format_stamp: "Formatted at 2026-08-10 23:25:33 on dist-test-slave-rmhg"
server_key: "adac8986cbf698e160f4c5508f6c6ea6"
server_key_iv: "44941b3a502b968168445fabe5730f83"
server_key_version: "encryptionkey@0"
I20260810 23:25:33.681456 20393 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260810 23:25:33.685530 20416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:33.687171 20393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:25:33.687369 20393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-2/wal
uuid: "d116956c15d34375898f83fcebccd32b"
format_stamp: "Formatted at 2026-08-10 23:25:33 on dist-test-slave-rmhg"
server_key: "adac8986cbf698e160f4c5508f6c6ea6"
server_key_iv: "44941b3a502b968168445fabe5730f83"
server_key_version: "encryptionkey@0"
I20260810 23:25:33.687597 20393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:33.708889 20393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:33.713574 20393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:33.713898 20393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:33.714917 20393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:25:33.716313 20393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:25:33.716396 20393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:33.716532 20393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:25:33.716588 20393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:33.724767 20192 heartbeater.cc:503] Master 127.14.175.126:45381 was elected leader, sending a full tablet report...
I20260810 23:25:33.748880 20393 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.67:44299
I20260810 23:25:33.748903 20590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.67:44299 every 8 connection(s)
I20260810 23:25:33.750310 20393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
I20260810 23:25:33.757786 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 20393
I20260810 23:25:33.758020 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance
I20260810 23:25:33.758582 15037 external_mini_cluster.cc:1428] Setting key 8786a3ace1dcb2cb4adeef7aa546448c
Aug 10 23:25:33 dist-test-slave-rmhg krb5kdc[19836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404333, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:33.772918 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:33.753208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:52803 (local address 127.14.175.126:45381)
0810 23:25:33.753681 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:33.753686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:33.754641 (+ 955us) server_negotiation.cc:407] Connection header received
0810 23:25:33.756533 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:33.756540 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:33.756628 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:33.756737 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:33.758911 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:33.759927 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:33.762245 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:33.762680 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:33.762782 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:33.766595 (+ 3813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:33.766620 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:33.766623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:33.766664 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:33.769349 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:33.770424 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:33.770428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:33.770431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:33.770561 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:33.771055 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:33.771059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:33.771061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:33.771375 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:33.771504 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:33.772270 (+ 766us) server_negotiation.cc:299] Negotiation successful
0810 23:25:33.772532 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":226,"threads_started":1}
I20260810 23:25:33.774034 20594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:33.753719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45381 (local address 127.14.175.67:52803)
0810 23:25:33.754461 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:33.754513 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:33.756219 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:33.757012 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:33.757021 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:33.757789 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:33.758688 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:33.758719 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:33.760116 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:33.760121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:33.762070 (+ 1949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:33.762085 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:33.762888 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:33.762894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:33.763119 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:33.763785 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:33.763815 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:33.766319 (+ 2504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:33.769524 (+ 3205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:33.769534 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:33.769550 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:33.770288 (+ 738us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:33.770687 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:33.770691 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:33.770694 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:33.770907 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:33.771510 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:33.771517 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:33.771921 (+ 404us) client_negotiation.cc:769] Sending connection context
0810 23:25:33.772181 (+ 260us) client_negotiation.cc:240] Negotiation successful
0810 23:25:33.772633 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":283,"threads_started":1}
I20260810 23:25:33.775527 20591 heartbeater.cc:347] Connected to a master server at 127.14.175.126:45381
I20260810 23:25:33.775841 20591 heartbeater.cc:464] Registering TS with master...
I20260810 23:25:33.776580 20591 heartbeater.cc:511] Master 127.14.175.126:45381 requested a full tablet report, sending...
I20260810 23:25:33.778219 19907 ts_manager.cc:194] Registered new tserver with Master: d116956c15d34375898f83fcebccd32b (127.14.175.67:44299)
I20260810 23:25:33.779074 19907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.67@KRBTEST.COM'} at 127.14.175.67:52803
I20260810 23:25:33.787694 15037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 23:25:33 dist-test-slave-rmhg krb5kdc[19836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404333, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 23:25:33 dist-test-slave-rmhg krb5kdc[19836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404333, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:33.842801 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:33.828048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58340 (local address 127.14.175.126:45381)
0810 23:25:33.828222 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:33.828227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:33.828681 (+ 454us) server_negotiation.cc:407] Connection header received
0810 23:25:33.828872 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:33.828876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:33.828945 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:33.829023 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:33.830494 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:33.831313 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:33.832598 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:33.833045 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:33.833180 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:33.837018 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:33.837024 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:33.837027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:33.837075 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:33.839939 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:33.840780 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:33.840785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:33.840787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:33.840941 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:33.841442 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:33.841446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:33.841448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:33.841853 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:33.842057 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:33.842328 (+ 271us) server_negotiation.cc:299] Negotiation successful
0810 23:25:33.842567 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260810 23:25:33.843418 19907 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58340
I20260810 23:25:33.845242 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 19987
Aug 10 23:25:33 dist-test-slave-rmhg krb5kdc[19836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404333, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.14.175.65@KRBTEST.COM
I20260810 23:25:33.865317 20606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:33.846290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54470 (local address 127.14.175.65:42687)
0810 23:25:33.849728 (+ 3438us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:33.849736 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:33.849765 (+ 29us) server_negotiation.cc:407] Connection header received
0810 23:25:33.849839 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:33.849845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:33.850076 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:33.850294 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:33.851631 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:33.852707 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:33.854031 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:33.854661 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:33.854791 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:33.859169 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:33.859194 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:33.859213 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:33.859256 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:33.862161 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:33.862964 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:33.862973 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:33.862976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:33.863123 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:33.863534 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:33.863538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:33.863541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:33.863928 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:33.864240 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:33.864580 (+ 340us) server_negotiation.cc:299] Negotiation successful
0810 23:25:33.864858 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3239,"thread_start_us":3086,"threads_started":1}
W20260810 23:25:33.866586 20182 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54470
W20260810 23:25:33.867767 15037 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 19987: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:25:33.867969 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 19987
I20260810 23:25:33.892688 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 20202
Aug 10 23:25:33 dist-test-slave-rmhg krb5kdc[19836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404333, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.14.175.66@KRBTEST.COM
I20260810 23:25:33.910014 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:33.893644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57772 (local address 127.14.175.66:34131)
0810 23:25:33.894228 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:33.894235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:33.894263 (+ 28us) server_negotiation.cc:407] Connection header received
0810 23:25:33.894324 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:33.894330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:33.894545 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:33.894705 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:33.895977 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:33.897005 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:33.898362 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:33.898965 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:33.899099 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:33.903516 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:33.903536 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:33.903556 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:33.903597 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:33.906776 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:33.907674 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:33.907683 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:33.907685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:33.907877 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:33.908436 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:33.908441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:33.908443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:33.908855 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:33.909216 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:33.909474 (+ 258us) server_negotiation.cc:299] Negotiation successful
0810 23:25:33.909719 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":298,"threads_started":1}
W20260810 23:25:33.911449 20377 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57772
W20260810 23:25:33.912612 15037 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 20202: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:25:33.912810 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 20202
I20260810 23:25:33.945349 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 20393
Aug 10 23:25:33 dist-test-slave-rmhg krb5kdc[19836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404333, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.14.175.67@KRBTEST.COM
I20260810 23:25:33.962682 20608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:33.946364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36494 (local address 127.14.175.67:44299)
0810 23:25:33.947078 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:33.947085 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:33.947118 (+ 33us) server_negotiation.cc:407] Connection header received
0810 23:25:33.947185 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:33.947192 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:33.947411 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:33.947574 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:33.949026 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:33.950210 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:33.951572 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:33.952181 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:33.952302 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:33.956815 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:33.956830 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:33.956848 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:33.956894 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:33.959666 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:33.960396 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:33.960405 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:33.960407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:33.960548 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:33.960926 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:33.960929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:33.960931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:33.961372 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:33.961682 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:33.962029 (+ 347us) server_negotiation.cc:299] Negotiation successful
0810 23:25:33.962322 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":357,"threads_started":1}
W20260810 23:25:33.964164 20585 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36494
W20260810 23:25:33.965502 15037 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 20393: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:25:33.965675 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 20393
I20260810 23:25:33.990180 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 19855
I20260810 23:25:33.991462 19949 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:25:34.097847 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 19855
2026-08-10T23:25:34Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3956 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:25:34 dist-test-slave-rmhg krb5kdc[20613](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:25:34 dist-test-slave-rmhg krb5kdc[20613](info): set up 2 sockets
Aug 10 23:25:34 dist-test-slave-rmhg krb5kdc[20613](info): commencing operation
krb5kdc: starting...
W20260810 23:25:35.324108 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.176s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:25:35 dist-test-slave-rmhg krb5kdc[20613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:25:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:25:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
I20260810 23:25:35.557811 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:32867
--webserver_interface=127.14.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:33303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:32867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:35.912302 20629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:35.912658 20629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:35.912783 20629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:35.920367 20629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:25:35.920507 20629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:35.920557 20629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:35.920701 20629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:25:35.920761 20629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:25:35.931104 20629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:33303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:32867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:32867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:35.932525 20629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:35.933921 20629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:35.945796 20638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:35.945762 20639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:35.945732 20644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:35.946717 20629 server_base.cc:1034] running on GCE node
I20260810 23:25:35.947561 20629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:35.949098 20629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:35.950353 20629 hybrid_clock.cc:648] HybridClock initialized: now 1786404335950284 us; error 47 us; skew 500 ppm
I20260810 23:25:35.950749 20629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:35 dist-test-slave-rmhg krb5kdc[20613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404335, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:35.953962 20629 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:25:35.955605 20629 webserver.cc:460] Webserver started at http://127.14.175.126:33157/ using document root <none> and password file <none>
I20260810 23:25:35.956300 20629 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:35.956406 20629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:35.956708 20629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:35.959956 20629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "8e8c780df1af4199955e700e3eb9bcdb"
format_stamp: "Formatted at 2026-08-10 23:25:35 on dist-test-slave-rmhg"
server_key: "ab1dbe347098fc668edb30050b51c396"
server_key_iv: "cbf1da518e01da26bf797b63728d9764"
server_key_version: "encryptionkey@0"
I20260810 23:25:35.960774 20629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "8e8c780df1af4199955e700e3eb9bcdb"
format_stamp: "Formatted at 2026-08-10 23:25:35 on dist-test-slave-rmhg"
server_key: "ab1dbe347098fc668edb30050b51c396"
server_key_iv: "cbf1da518e01da26bf797b63728d9764"
server_key_version: "encryptionkey@0"
I20260810 23:25:35.966394 20629 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 23:25:35.970118 20656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:35.971623 20629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:25:35.971788 20629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "8e8c780df1af4199955e700e3eb9bcdb"
format_stamp: "Formatted at 2026-08-10 23:25:35 on dist-test-slave-rmhg"
server_key: "ab1dbe347098fc668edb30050b51c396"
server_key_iv: "cbf1da518e01da26bf797b63728d9764"
server_key_version: "encryptionkey@0"
I20260810 23:25:35.971968 20629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:35.988799 20629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:35.993273 20629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:35.993532 20629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:36.012190 20629 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:32867
I20260810 23:25:36.012200 20726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:32867 every 8 connection(s)
I20260810 23:25:36.013703 20629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:25:36.017918 20728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:36.021560 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 20629
I20260810 23:25:36.021797 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:25:36.022591 15037 external_mini_cluster.cc:1428] Setting key 8137941e5ab2d64ca4f11a2f217be9bc
I20260810 23:25:36.031955 20728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e8c780df1af4199955e700e3eb9bcdb: Bootstrap starting.
I20260810 23:25:36.036166 20728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e8c780df1af4199955e700e3eb9bcdb: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:36.037626 20728 log.cc:824] T 00000000000000000000000000000000 P 8e8c780df1af4199955e700e3eb9bcdb: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:25:36 dist-test-slave-rmhg krb5kdc[20613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:36.040876 20728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e8c780df1af4199955e700e3eb9bcdb: No bootstrap required, opened a new log
I20260810 23:25:36.047186 20734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:36.024920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39252 (local address 127.14.175.126:32867)
0810 23:25:36.025822 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:36.025836 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:36.025882 (+ 46us) server_negotiation.cc:407] Connection header received
0810 23:25:36.027617 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:36.027647 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:36.028157 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:36.028537 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:36.030496 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:36.031906 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:36.033769 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:36.034532 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:36.034697 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:36.039522 (+ 4825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:36.039530 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:36.039547 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:36.039584 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:36.042683 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:36.043586 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:36.043593 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:36.043599 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:36.043810 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:36.044207 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:36.044211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:36.044213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:36.044860 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:36.045079 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:36.045599 (+ 520us) server_negotiation.cc:299] Negotiation successful
0810 23:25:36.046072 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":432,"threads_started":1}
I20260810 23:25:36.050053 20728 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e8c780df1af4199955e700e3eb9bcdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8c780df1af4199955e700e3eb9bcdb" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 32867 } }
I20260810 23:25:36.050508 20728 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e8c780df1af4199955e700e3eb9bcdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:36.050618 20728 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e8c780df1af4199955e700e3eb9bcdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e8c780df1af4199955e700e3eb9bcdb, State: Initialized, Role: FOLLOWER
I20260810 23:25:36.051173 20728 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e8c780df1af4199955e700e3eb9bcdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8c780df1af4199955e700e3eb9bcdb" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 32867 } }
I20260810 23:25:36.051409 20728 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e8c780df1af4199955e700e3eb9bcdb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:25:36.051538 20728 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e8c780df1af4199955e700e3eb9bcdb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:25:36.051702 20728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e8c780df1af4199955e700e3eb9bcdb [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:36.053968 20728 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e8c780df1af4199955e700e3eb9bcdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8c780df1af4199955e700e3eb9bcdb" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 32867 } }
I20260810 23:25:36.054474 20728 leader_election.cc:302] T 00000000000000000000000000000000 P 8e8c780df1af4199955e700e3eb9bcdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e8c780df1af4199955e700e3eb9bcdb; no voters:
I20260810 23:25:36.055058 20728 leader_election.cc:288] T 00000000000000000000000000000000 P 8e8c780df1af4199955e700e3eb9bcdb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:25:36.055195 20739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e8c780df1af4199955e700e3eb9bcdb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:25:36.055954 20739 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e8c780df1af4199955e700e3eb9bcdb [term 1 LEADER]: Becoming Leader. State: Replica: 8e8c780df1af4199955e700e3eb9bcdb, State: Running, Role: LEADER
I20260810 23:25:36.056660 20739 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e8c780df1af4199955e700e3eb9bcdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8c780df1af4199955e700e3eb9bcdb" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 32867 } }
I20260810 23:25:36.057463 20728 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e8c780df1af4199955e700e3eb9bcdb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:25:36.059522 20741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e8c780df1af4199955e700e3eb9bcdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e8c780df1af4199955e700e3eb9bcdb. Latest consensus state: current_term: 1 leader_uuid: "8e8c780df1af4199955e700e3eb9bcdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8c780df1af4199955e700e3eb9bcdb" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 32867 } } }
I20260810 23:25:36.059558 20740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e8c780df1af4199955e700e3eb9bcdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e8c780df1af4199955e700e3eb9bcdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8c780df1af4199955e700e3eb9bcdb" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 32867 } } }
I20260810 23:25:36.060065 20740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e8c780df1af4199955e700e3eb9bcdb [sys.catalog]: This master's current role is: LEADER
I20260810 23:25:36.060658 20741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e8c780df1af4199955e700e3eb9bcdb [sys.catalog]: This master's current role is: LEADER
I20260810 23:25:36.061661 20746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:25:36.067191 20746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:25:36.077495 20746 catalog_manager.cc:1347] Generated new cluster ID: e76044679c0d432a8ba9ea2f64e105b5
I20260810 23:25:36.077620 20746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:25:36.101465 20746 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:25:36.102945 20746 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:25:36.111280 20746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e8c780df1af4199955e700e3eb9bcdb: Generated new TSK 0
I20260810 23:25:36.112215 20746 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
I20260810 23:25:36.178128 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.65:0
--local_ip_for_outbound_sockets=127.14.175.65
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:32867
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:33303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:36.558863 20764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:36.559165 20764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:36.559278 20764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:36.566948 20764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:36.567086 20764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:36.567258 20764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.65
I20260810 23:25:36.577848 20764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:33303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:32867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.65
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:36.579282 20764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:36.580674 20764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:36.594548 20774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:36.594559 20771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:36.594718 20776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:36.595496 20764 server_base.cc:1034] running on GCE node
I20260810 23:25:36.596186 20764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:36.597229 20764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:36.598455 20764 hybrid_clock.cc:648] HybridClock initialized: now 1786404336598393 us; error 54 us; skew 500 ppm
I20260810 23:25:36.598824 20764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:36 dist-test-slave-rmhg krb5kdc[20613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404336, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:36.602180 20764 init.cc:377] Logged in from keytab as kudu/127.14.175.65@KRBTEST.COM (short username kudu)
I20260810 23:25:36.603911 20764 webserver.cc:460] Webserver started at http://127.14.175.65:36353/ using document root <none> and password file <none>
I20260810 23:25:36.604678 20764 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:36.604794 20764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:36.605065 20764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:36.608254 20764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-0/data/instance:
uuid: "46d4cd2d6f65430fbe0f565a3b24b6bd"
format_stamp: "Formatted at 2026-08-10 23:25:36 on dist-test-slave-rmhg"
server_key: "562f4a17c39f9d5131d135ff2997dfa7"
server_key_iv: "310e3663b6913440cb9bdea8e6f9cb0c"
server_key_version: "encryptionkey@0"
I20260810 23:25:36.609066 20764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance:
uuid: "46d4cd2d6f65430fbe0f565a3b24b6bd"
format_stamp: "Formatted at 2026-08-10 23:25:36 on dist-test-slave-rmhg"
server_key: "562f4a17c39f9d5131d135ff2997dfa7"
server_key_iv: "310e3663b6913440cb9bdea8e6f9cb0c"
server_key_version: "encryptionkey@0"
I20260810 23:25:36.614758 20764 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 23:25:36.618714 20785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:36.620242 20764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:25:36.620421 20764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-0/wal
uuid: "46d4cd2d6f65430fbe0f565a3b24b6bd"
format_stamp: "Formatted at 2026-08-10 23:25:36 on dist-test-slave-rmhg"
server_key: "562f4a17c39f9d5131d135ff2997dfa7"
server_key_iv: "310e3663b6913440cb9bdea8e6f9cb0c"
server_key_version: "encryptionkey@0"
I20260810 23:25:36.620638 20764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:36.639659 20764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:36.644516 20764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:36.644850 20764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:36.646049 20764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:25:36.647748 20764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:25:36.647835 20764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:36.647933 20764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:25:36.647976 20764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:36.681735 20764 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.65:40061
I20260810 23:25:36.681811 20955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.65:40061 every 8 connection(s)
I20260810 23:25:36.683145 20764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
I20260810 23:25:36.687536 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 20764
I20260810 23:25:36.687769 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance
I20260810 23:25:36.688642 15037 external_mini_cluster.cc:1428] Setting key 7c05603de9b5b77b1bfb1fd503bdf58d
Aug 10 23:25:36 dist-test-slave-rmhg krb5kdc[20613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404336, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:36.707254 20961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:36.685836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:41899 (local address 127.14.175.126:32867)
0810 23:25:36.686282 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:36.686287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:36.687399 (+ 1112us) server_negotiation.cc:407] Connection header received
0810 23:25:36.689904 (+ 2505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:36.689909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:36.689988 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:36.690123 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:36.692143 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:36.692886 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:36.695194 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:36.695646 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:36.695771 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:36.700483 (+ 4712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:36.700489 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:36.700493 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:36.700535 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:36.703282 (+ 2747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:36.704396 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:36.704401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:36.704403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:36.704561 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:36.705113 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:36.705117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:36.705119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:36.705495 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:36.705643 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:36.706760 (+ 1117us) server_negotiation.cc:299] Negotiation successful
0810 23:25:36.706973 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":238,"threads_started":1}
I20260810 23:25:36.708806 20963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:36.686341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:32867 (local address 127.14.175.65:41899)
0810 23:25:36.687159 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:36.687231 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:36.689582 (+ 2351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:36.690335 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:36.690345 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:36.691064 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:36.691943 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:36.691971 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:36.693060 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:36.693065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:36.694984 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:36.695002 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:36.695919 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:36.695927 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:36.696242 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:36.697045 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:36.697082 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:36.700172 (+ 3090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:36.703501 (+ 3329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:36.703522 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:36.703546 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:36.704242 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:36.704692 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:36.704697 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:36.704700 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:36.704987 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:36.705670 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:36.705679 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:36.706241 (+ 562us) client_negotiation.cc:769] Sending connection context
0810 23:25:36.706648 (+ 407us) client_negotiation.cc:240] Negotiation successful
0810 23:25:36.707255 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":290,"threads_started":1}
I20260810 23:25:36.710791 20956 heartbeater.cc:347] Connected to a master server at 127.14.175.126:32867
I20260810 23:25:36.711143 20956 heartbeater.cc:464] Registering TS with master...
I20260810 23:25:36.711949 20956 heartbeater.cc:511] Master 127.14.175.126:32867 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.175.66@KRBTEST.COM; defaulting to no policy
I20260810 23:25:36.714649 20685 ts_manager.cc:194] Registered new tserver with Master: 46d4cd2d6f65430fbe0f565a3b24b6bd (127.14.175.65:40061)
I20260810 23:25:36.716349 20685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.65@KRBTEST.COM'} at 127.14.175.65:41899
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
I20260810 23:25:36.745457 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.66:0
--local_ip_for_outbound_sockets=127.14.175.66
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:32867
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:33303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:37.094578 20968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:37.094928 20968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:37.095055 20968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:37.102959 20968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:37.103124 20968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:37.103293 20968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.66
I20260810 23:25:37.113759 20968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:33303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:32867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.66
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:37.115221 20968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:37.116637 20968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:37.131469 20977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:37.131534 20978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:37.131492 20981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:37.132162 20968 server_base.cc:1034] running on GCE node
I20260810 23:25:37.132980 20968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:37.134097 20968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:37.135346 20968 hybrid_clock.cc:648] HybridClock initialized: now 1786404337135298 us; error 43 us; skew 500 ppm
I20260810 23:25:37.135735 20968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:37 dist-test-slave-rmhg krb5kdc[20613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404337, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:37.144320 20968 init.cc:377] Logged in from keytab as kudu/127.14.175.66@KRBTEST.COM (short username kudu)
I20260810 23:25:37.146749 20968 webserver.cc:460] Webserver started at http://127.14.175.66:36865/ using document root <none> and password file <none>
I20260810 23:25:37.147540 20968 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:37.147671 20968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:37.148006 20968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:37.151255 20968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-1/data/instance:
uuid: "080f7816993b4f17b499158dfef8ec8b"
format_stamp: "Formatted at 2026-08-10 23:25:37 on dist-test-slave-rmhg"
server_key: "0a0ef1d7084f92fab1f0a7a4206d03e1"
server_key_iv: "10a05dac14d8e03e5d97650f88af412d"
server_key_version: "encryptionkey@0"
I20260810 23:25:37.152098 20968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance:
uuid: "080f7816993b4f17b499158dfef8ec8b"
format_stamp: "Formatted at 2026-08-10 23:25:37 on dist-test-slave-rmhg"
server_key: "0a0ef1d7084f92fab1f0a7a4206d03e1"
server_key_iv: "10a05dac14d8e03e5d97650f88af412d"
server_key_version: "encryptionkey@0"
I20260810 23:25:37.158309 20968 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260810 23:25:37.162776 20992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:37.164503 20968 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260810 23:25:37.164757 20968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-1/wal
uuid: "080f7816993b4f17b499158dfef8ec8b"
format_stamp: "Formatted at 2026-08-10 23:25:37 on dist-test-slave-rmhg"
server_key: "0a0ef1d7084f92fab1f0a7a4206d03e1"
server_key_iv: "10a05dac14d8e03e5d97650f88af412d"
server_key_version: "encryptionkey@0"
I20260810 23:25:37.164996 20968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:37.181527 20968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:37.186560 20968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:37.186872 20968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:37.187953 20968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:25:37.189512 20968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:25:37.189615 20968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:37.189747 20968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:25:37.189806 20968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:37.222157 20968 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.66:33757
I20260810 23:25:37.222205 21148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.66:33757 every 8 connection(s)
I20260810 23:25:37.223595 20968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
I20260810 23:25:37.230041 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 20968
I20260810 23:25:37.230230 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance
I20260810 23:25:37.230680 15037 external_mini_cluster.cc:1428] Setting key 2024dbfd2265b8d09bda8d8e0a4729cb
Aug 10 23:25:37 dist-test-slave-rmhg krb5kdc[20613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404337, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:37.247066 21152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:37.226508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:40009 (local address 127.14.175.126:32867)
0810 23:25:37.226975 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:37.226979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:37.227998 (+ 1019us) server_negotiation.cc:407] Connection header received
0810 23:25:37.230020 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:37.230026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:37.230116 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:37.230217 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:37.232024 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:37.232921 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:37.235411 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:37.235886 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:37.235989 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:37.240330 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:37.240337 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:37.240341 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:37.240385 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:37.243475 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:37.244439 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:37.244443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:37.244446 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:37.244598 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:37.245234 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:37.245238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:37.245241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:37.245537 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:37.245684 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:37.246576 (+ 892us) server_negotiation.cc:299] Negotiation successful
0810 23:25:37.246797 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":255,"threads_started":1}
I20260810 23:25:37.248495 21153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:37.227047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:32867 (local address 127.14.175.66:40009)
0810 23:25:37.227825 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:37.227869 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:37.229705 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:37.230502 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:37.230513 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:37.231093 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:37.231844 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:37.231865 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:37.233143 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:37.233173 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:37.235248 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:37.235267 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:37.236145 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:37.236153 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:37.236468 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:37.237280 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:37.237312 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:37.240011 (+ 2699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:37.243662 (+ 3651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:37.243681 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:37.243701 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:37.244321 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:37.244775 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:37.244780 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:37.244784 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:37.245079 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:37.245667 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:37.245684 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:37.246153 (+ 469us) client_negotiation.cc:769] Sending connection context
0810 23:25:37.246463 (+ 310us) client_negotiation.cc:240] Negotiation successful
0810 23:25:37.247105 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":292,"threads_started":1}
I20260810 23:25:37.250191 21149 heartbeater.cc:347] Connected to a master server at 127.14.175.126:32867
I20260810 23:25:37.250554 21149 heartbeater.cc:464] Registering TS with master...
I20260810 23:25:37.251508 21149 heartbeater.cc:511] Master 127.14.175.126:32867 requested a full tablet report, sending...
I20260810 23:25:37.253309 20685 ts_manager.cc:194] Registered new tserver with Master: 080f7816993b4f17b499158dfef8ec8b (127.14.175.66:33757)
I20260810 23:25:37.254490 20685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.66@KRBTEST.COM'} at 127.14.175.66:40009
WARNING: no policy specified for kudu/127.14.175.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
I20260810 23:25:37.284971 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.67:0
--local_ip_for_outbound_sockets=127.14.175.67
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:32867
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:33303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:37.634200 21157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:37.634517 21157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:37.634624 21157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:37.642256 21157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:37.642395 21157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:37.642513 21157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.67
I20260810 23:25:37.652861 21157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:33303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:32867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.67
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:37.654414 21157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:37.655771 21157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:37.669829 21167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:37.669870 21165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:37.669993 21163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:37.671239 21157 server_base.cc:1034] running on GCE node
I20260810 23:25:37.671974 21157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:37.673063 21157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:37.674337 21157 hybrid_clock.cc:648] HybridClock initialized: now 1786404337674263 us; error 48 us; skew 500 ppm
I20260810 23:25:37.674752 21157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:37 dist-test-slave-rmhg krb5kdc[20613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404337, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:37.678587 21157 init.cc:377] Logged in from keytab as kudu/127.14.175.67@KRBTEST.COM (short username kudu)
I20260810 23:25:37.680320 21157 webserver.cc:460] Webserver started at http://127.14.175.67:35175/ using document root <none> and password file <none>
I20260810 23:25:37.681116 21157 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:37.681304 21157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:37.681653 21157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:37.684841 21157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-2/data/instance:
uuid: "c1d60187eabd4783ba7de074d07ba258"
format_stamp: "Formatted at 2026-08-10 23:25:37 on dist-test-slave-rmhg"
server_key: "2e06ada38ef27b85182e80b960147193"
server_key_iv: "990bc7b07b86720fb35ee8aaf42a115a"
server_key_version: "encryptionkey@0"
I20260810 23:25:37.685802 21157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance:
uuid: "c1d60187eabd4783ba7de074d07ba258"
format_stamp: "Formatted at 2026-08-10 23:25:37 on dist-test-slave-rmhg"
server_key: "2e06ada38ef27b85182e80b960147193"
server_key_iv: "990bc7b07b86720fb35ee8aaf42a115a"
server_key_version: "encryptionkey@0"
I20260810 23:25:37.691659 21157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:25:37.695798 21179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:37.697492 21157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:25:37.697734 21157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-2/wal
uuid: "c1d60187eabd4783ba7de074d07ba258"
format_stamp: "Formatted at 2026-08-10 23:25:37 on dist-test-slave-rmhg"
server_key: "2e06ada38ef27b85182e80b960147193"
server_key_iv: "990bc7b07b86720fb35ee8aaf42a115a"
server_key_version: "encryptionkey@0"
I20260810 23:25:37.698144 21157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:37.720631 20956 heartbeater.cc:503] Master 127.14.175.126:32867 was elected leader, sending a full tablet report...
I20260810 23:25:37.728803 21157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:37.733692 21157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:37.734020 21157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:37.735097 21157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:25:37.736558 21157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:25:37.736644 21157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:37.736769 21157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:25:37.736825 21157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:37.769443 21157 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.67:36537
I20260810 23:25:37.769495 21346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.67:36537 every 8 connection(s)
I20260810 23:25:37.770978 21157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
I20260810 23:25:37.779743 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 21157
I20260810 23:25:37.779949 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance
I20260810 23:25:37.780522 15037 external_mini_cluster.cc:1428] Setting key 042c8789a4d851af3204aa934a3e5bb9
Aug 10 23:25:37 dist-test-slave-rmhg krb5kdc[20613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404337, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:37.793944 21350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:37.773703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:50915 (local address 127.14.175.126:32867)
0810 23:25:37.774191 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:37.774197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:37.775321 (+ 1124us) server_negotiation.cc:407] Connection header received
0810 23:25:37.777448 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:37.777453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:37.777548 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:37.777656 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:37.779452 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:37.780698 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:37.782983 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:37.783422 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:37.783541 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:37.787458 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:37.787464 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:37.787467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:37.787508 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:37.790338 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:37.791246 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:37.791251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:37.791253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:37.791397 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:37.791891 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:37.791895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:37.791898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:37.792235 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:37.792392 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:37.793284 (+ 892us) server_negotiation.cc:299] Negotiation successful
0810 23:25:37.793556 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":209,"threads_started":1}
I20260810 23:25:37.795100 21352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:37.774353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:32867 (local address 127.14.175.67:50915)
0810 23:25:37.775145 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:37.775196 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:37.777037 (+ 1841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:37.777904 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:37.777914 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:37.778515 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:37.779184 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:37.779205 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:37.780900 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:37.780904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:37.782817 (+ 1913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:37.782835 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:37.783685 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:37.783694 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:37.784005 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:37.784771 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:37.784798 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:37.787182 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:37.790540 (+ 3358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:37.790550 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:37.790568 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:37.791082 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:37.791527 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:37.791531 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:37.791534 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:37.791751 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:37.792404 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:37.792414 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:37.792820 (+ 406us) client_negotiation.cc:769] Sending connection context
0810 23:25:37.793104 (+ 284us) client_negotiation.cc:240] Negotiation successful
0810 23:25:37.793767 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":288,"threads_started":1}
I20260810 23:25:37.796649 21348 heartbeater.cc:347] Connected to a master server at 127.14.175.126:32867
I20260810 23:25:37.797030 21348 heartbeater.cc:464] Registering TS with master...
I20260810 23:25:37.797842 21348 heartbeater.cc:511] Master 127.14.175.126:32867 requested a full tablet report, sending...
I20260810 23:25:37.799374 20685 ts_manager.cc:194] Registered new tserver with Master: c1d60187eabd4783ba7de074d07ba258 (127.14.175.67:36537)
I20260810 23:25:37.800233 20685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.67@KRBTEST.COM'} at 127.14.175.67:50915
I20260810 23:25:37.809902 15037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 23:25:37 dist-test-slave-rmhg krb5kdc[20613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 23:25:37 dist-test-slave-rmhg krb5kdc[20613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.65@KRBTEST.COM
I20260810 23:25:37.865375 21359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:37.849470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47864 (local address 127.14.175.65:40061)
0810 23:25:37.850134 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:37.850144 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:37.850175 (+ 31us) server_negotiation.cc:407] Connection header received
0810 23:25:37.850238 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:37.850245 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:37.850463 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:37.850620 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:37.852142 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:37.853213 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:37.854460 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:37.855063 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:37.855193 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:37.859215 (+ 4022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:37.859229 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:37.859246 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:37.859284 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:37.862326 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:37.863079 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:37.863087 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:37.863090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:37.863244 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:37.863725 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:37.863730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:37.863733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:37.864201 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:37.864449 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:37.864792 (+ 343us) server_negotiation.cc:299] Negotiation successful
0810 23:25:37.865010 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":309,"threads_started":1}
Aug 10 23:25:37 dist-test-slave-rmhg krb5kdc[20613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:37.884735 21350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:37.870510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39264 (local address 127.14.175.126:32867)
0810 23:25:37.870736 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:37.870742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:37.870826 (+ 84us) server_negotiation.cc:407] Connection header received
0810 23:25:37.871006 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:37.871012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:37.871086 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:37.871172 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:37.872470 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:37.873330 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:37.874610 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:37.875041 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:37.875161 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:37.879385 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:37.879391 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:37.879394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:37.879434 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:37.882040 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:37.882753 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:37.882758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:37.882761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:37.882891 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:37.883253 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:37.883257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:37.883260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:37.883609 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:37.883858 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:37.884171 (+ 313us) server_negotiation.cc:299] Negotiation successful
0810 23:25:37.884404 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260810 23:25:37.885483 20685 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39264
I20260810 23:25:37.886554 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 20764
I20260810 23:25:37.900074 21359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:37.887481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47868 (local address 127.14.175.65:40061)
0810 23:25:37.887707 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:37.887714 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:37.887953 (+ 239us) server_negotiation.cc:407] Connection header received
0810 23:25:37.888081 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:37.888085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:37.888187 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:37.888269 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:37.889628 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:37.890355 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:37.891578 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:37.891987 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:37.892086 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:37.894589 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:37.894595 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:37.894599 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:37.894656 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:37.897532 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:37.898344 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:37.898350 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:37.898352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:37.898520 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:37.898842 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:37.898845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:37.898847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:37.899131 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:37.899319 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:37.899583 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 23:25:37.899773 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 23:25:37.900719 20949 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:25:37.984449 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 20764
I20260810 23:25:38.009594 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 20968
Aug 10 23:25:38 dist-test-slave-rmhg krb5kdc[20613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.66@KRBTEST.COM
I20260810 23:25:38.026429 21368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:38.010518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53858 (local address 127.14.175.66:33757)
0810 23:25:38.011100 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:38.011108 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:38.011137 (+ 29us) server_negotiation.cc:407] Connection header received
0810 23:25:38.011202 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:38.011208 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:38.011422 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:38.011590 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:38.012884 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:38.013968 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:38.015269 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:38.015889 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:38.016017 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:38.020273 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:38.020290 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:38.020308 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:38.020350 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:38.023313 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:38.024053 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:38.024061 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:38.024064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:38.024206 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:38.024611 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:38.024615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:38.024618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:38.025033 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:38.025383 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:38.025811 (+ 428us) server_negotiation.cc:299] Negotiation successful
0810 23:25:38.026084 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":315,"threads_started":1}
I20260810 23:25:38.027819 21143 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:25:38.113057 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 20968
I20260810 23:25:38.142273 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 21157
Aug 10 23:25:38 dist-test-slave-rmhg krb5kdc[20613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.67@KRBTEST.COM
I20260810 23:25:38.162494 21370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:38.143374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39416 (local address 127.14.175.67:36537)
0810 23:25:38.144074 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:38.144082 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:38.144115 (+ 33us) server_negotiation.cc:407] Connection header received
0810 23:25:38.144177 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:38.144185 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:38.144404 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:38.144571 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:38.146163 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:38.147667 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:38.149499 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:38.150255 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:38.150393 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:38.155212 (+ 4819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:38.155232 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:38.155250 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:38.155298 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:38.158985 (+ 3687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:38.159930 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:38.159941 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:38.159943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:38.160134 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:38.160604 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:38.160609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:38.160612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:38.161034 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:38.161435 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:38.161818 (+ 383us) server_negotiation.cc:299] Negotiation successful
0810 23:25:38.162139 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":342,"threads_started":1}
I20260810 23:25:38.164052 21338 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:25:38.249338 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 21157
I20260810 23:25:38.283493 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 20629
I20260810 23:25:38.284857 20720 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:25:38.395145 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 20629
2026-08-10T23:25:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4298 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:25:38 dist-test-slave-rmhg krb5kdc[21380](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:25:38 dist-test-slave-rmhg krb5kdc[21380](info): set up 2 sockets
Aug 10 23:25:38 dist-test-slave-rmhg krb5kdc[21380](info): commencing operation
krb5kdc: starting...
W20260810 23:25:39.622444 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.176s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:25:39 dist-test-slave-rmhg krb5kdc[21380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:25:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:25:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
I20260810 23:25:39.860528 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:33851
--webserver_interface=127.14.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:44133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:33851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:40.211455 21397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:40.211790 21397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:40.211910 21397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:40.219444 21397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:25:40.219584 21397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:40.219638 21397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:40.219717 21397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:25:40.219785 21397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:25:40.230494 21397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:44133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:33851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:33851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:40.231926 21397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:40.233359 21397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:40.245232 21407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:40.245271 21405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:40.245564 21404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:40.246320 21397 server_base.cc:1034] running on GCE node
I20260810 23:25:40.247191 21397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:40.248690 21397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:40.249917 21397 hybrid_clock.cc:648] HybridClock initialized: now 1786404340249855 us; error 49 us; skew 500 ppm
I20260810 23:25:40.250310 21397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:40 dist-test-slave-rmhg krb5kdc[21380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404340, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:40.253648 21397 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:25:40.254900 21397 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:40.255003 21397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:40.255366 21397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:40.258665 21397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "2d0627788d1344f2a1f112c190326655"
format_stamp: "Formatted at 2026-08-10 23:25:40 on dist-test-slave-rmhg"
server_key: "0f7b6e5b7421f77ecad9e28282523487"
server_key_iv: "c2d370dbd9d8877cacb0e099e3db2069"
server_key_version: "encryptionkey@0"
I20260810 23:25:40.259505 21397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "2d0627788d1344f2a1f112c190326655"
format_stamp: "Formatted at 2026-08-10 23:25:40 on dist-test-slave-rmhg"
server_key: "0f7b6e5b7421f77ecad9e28282523487"
server_key_iv: "c2d370dbd9d8877cacb0e099e3db2069"
server_key_version: "encryptionkey@0"
I20260810 23:25:40.265038 21397 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 23:25:40.268978 21415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:40.270496 21397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:25:40.270676 21397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "2d0627788d1344f2a1f112c190326655"
format_stamp: "Formatted at 2026-08-10 23:25:40 on dist-test-slave-rmhg"
server_key: "0f7b6e5b7421f77ecad9e28282523487"
server_key_iv: "c2d370dbd9d8877cacb0e099e3db2069"
server_key_version: "encryptionkey@0"
I20260810 23:25:40.270908 21397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:40.293071 21397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:40.297894 21397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:40.298156 21397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:40.317309 21397 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:33851
I20260810 23:25:40.317291 21484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:33851 every 8 connection(s)
I20260810 23:25:40.318317 21397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:25:40.322407 21485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:40.324962 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 21397
I20260810 23:25:40.325219 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:25:40.325863 15037 external_mini_cluster.cc:1428] Setting key 255144715e0bdd54e0f3c8a8a8781ead
I20260810 23:25:40.335466 21485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d0627788d1344f2a1f112c190326655: Bootstrap starting.
I20260810 23:25:40.339717 21485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d0627788d1344f2a1f112c190326655: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:40.341209 21485 log.cc:824] T 00000000000000000000000000000000 P 2d0627788d1344f2a1f112c190326655: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:25:40 dist-test-slave-rmhg krb5kdc[21380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:40.344537 21485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d0627788d1344f2a1f112c190326655: No bootstrap required, opened a new log
I20260810 23:25:40.351176 21491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:40.328180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41112 (local address 127.14.175.126:33851)
0810 23:25:40.329086 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:40.329100 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:40.329183 (+ 83us) server_negotiation.cc:407] Connection header received
0810 23:25:40.330957 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:40.330992 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:40.331462 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:40.331847 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:40.333916 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:40.335478 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:40.337416 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:40.338183 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:40.338334 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:40.343478 (+ 5144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:40.343487 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:40.343504 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:40.343547 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:40.346729 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:40.347647 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:40.347654 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:40.347665 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:40.347849 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:40.348258 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:40.348261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:40.348263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:40.348876 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:40.349112 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:40.349630 (+ 518us) server_negotiation.cc:299] Negotiation successful
0810 23:25:40.350073 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":405,"threads_started":1}
I20260810 23:25:40.353798 21485 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d0627788d1344f2a1f112c190326655 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d0627788d1344f2a1f112c190326655" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 33851 } }
I20260810 23:25:40.354199 21485 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d0627788d1344f2a1f112c190326655 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:40.354318 21485 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d0627788d1344f2a1f112c190326655 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d0627788d1344f2a1f112c190326655, State: Initialized, Role: FOLLOWER
I20260810 23:25:40.354882 21485 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d0627788d1344f2a1f112c190326655 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d0627788d1344f2a1f112c190326655" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 33851 } }
I20260810 23:25:40.355094 21485 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d0627788d1344f2a1f112c190326655 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:25:40.355196 21485 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d0627788d1344f2a1f112c190326655 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:25:40.355322 21485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d0627788d1344f2a1f112c190326655 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:40.357532 21485 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d0627788d1344f2a1f112c190326655 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d0627788d1344f2a1f112c190326655" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 33851 } }
I20260810 23:25:40.358019 21485 leader_election.cc:302] T 00000000000000000000000000000000 P 2d0627788d1344f2a1f112c190326655 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d0627788d1344f2a1f112c190326655; no voters:
I20260810 23:25:40.358677 21485 leader_election.cc:288] T 00000000000000000000000000000000 P 2d0627788d1344f2a1f112c190326655 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:25:40.358830 21493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d0627788d1344f2a1f112c190326655 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:25:40.359567 21493 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d0627788d1344f2a1f112c190326655 [term 1 LEADER]: Becoming Leader. State: Replica: 2d0627788d1344f2a1f112c190326655, State: Running, Role: LEADER
I20260810 23:25:40.360169 21493 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d0627788d1344f2a1f112c190326655 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d0627788d1344f2a1f112c190326655" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 33851 } }
I20260810 23:25:40.361024 21485 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d0627788d1344f2a1f112c190326655 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:25:40.363305 21495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d0627788d1344f2a1f112c190326655 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d0627788d1344f2a1f112c190326655. Latest consensus state: current_term: 1 leader_uuid: "2d0627788d1344f2a1f112c190326655" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d0627788d1344f2a1f112c190326655" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 33851 } } }
I20260810 23:25:40.363649 21495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d0627788d1344f2a1f112c190326655 [sys.catalog]: This master's current role is: LEADER
I20260810 23:25:40.368254 21494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d0627788d1344f2a1f112c190326655 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d0627788d1344f2a1f112c190326655" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d0627788d1344f2a1f112c190326655" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 33851 } } }
I20260810 23:25:40.368537 21494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d0627788d1344f2a1f112c190326655 [sys.catalog]: This master's current role is: LEADER
I20260810 23:25:40.369083 21505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:25:40.374972 21505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:25:40.396013 21505 catalog_manager.cc:1347] Generated new cluster ID: fc4143cfaf1a497b85ff34f35c2853bf
I20260810 23:25:40.396162 21505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:25:40.421571 21505 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:25:40.423003 21505 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:25:40.432668 21505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d0627788d1344f2a1f112c190326655: Generated new TSK 0
I20260810 23:25:40.433668 21505 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
I20260810 23:25:40.504944 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.65:0
--local_ip_for_outbound_sockets=127.14.175.65
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:33851
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:44133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:40.911965 21522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:40.912303 21522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:40.912432 21522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:40.920009 21522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:40.920153 21522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:40.920315 21522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.65
I20260810 23:25:40.931085 21522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:44133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:33851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.65
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:40.932569 21522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:40.934106 21522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:40.948504 21529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:40.948522 21533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:40.948550 21531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:40.949323 21522 server_base.cc:1034] running on GCE node
I20260810 23:25:40.950038 21522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:40.951115 21522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:40.952333 21522 hybrid_clock.cc:648] HybridClock initialized: now 1786404340952269 us; error 54 us; skew 500 ppm
I20260810 23:25:40.952723 21522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:40 dist-test-slave-rmhg krb5kdc[21380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404340, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:40.956102 21522 init.cc:377] Logged in from keytab as kudu/127.14.175.65@KRBTEST.COM (short username kudu)
I20260810 23:25:40.957517 21522 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:40.957651 21522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:40.958004 21522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:40.961488 21522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-0/data/instance:
uuid: "689c75c2475d4978b0680bdf6c01711b"
format_stamp: "Formatted at 2026-08-10 23:25:40 on dist-test-slave-rmhg"
server_key: "865aae9e7ae10ed8d022873840430fcd"
server_key_iv: "c39f6179f03979b6f46540bc1c095b63"
server_key_version: "encryptionkey@0"
I20260810 23:25:40.962528 21522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance:
uuid: "689c75c2475d4978b0680bdf6c01711b"
format_stamp: "Formatted at 2026-08-10 23:25:40 on dist-test-slave-rmhg"
server_key: "865aae9e7ae10ed8d022873840430fcd"
server_key_iv: "c39f6179f03979b6f46540bc1c095b63"
server_key_version: "encryptionkey@0"
I20260810 23:25:40.968436 21522 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260810 23:25:40.972796 21542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:40.974416 21522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:25:40.974620 21522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-0/wal
uuid: "689c75c2475d4978b0680bdf6c01711b"
format_stamp: "Formatted at 2026-08-10 23:25:40 on dist-test-slave-rmhg"
server_key: "865aae9e7ae10ed8d022873840430fcd"
server_key_iv: "c39f6179f03979b6f46540bc1c095b63"
server_key_version: "encryptionkey@0"
I20260810 23:25:40.974891 21522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:40.990530 21522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:40.995404 21522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:40.995723 21522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:40.996829 21522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:25:40.998528 21522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:25:40.998641 21522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:40.998757 21522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:25:40.998831 21522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:41.032145 21522 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.65:45747
I20260810 23:25:41.032195 21713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.65:45747 every 8 connection(s)
I20260810 23:25:41.033082 21522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
I20260810 23:25:41.033576 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 21522
I20260810 23:25:41.033788 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance
I20260810 23:25:41.034659 15037 external_mini_cluster.cc:1428] Setting key ac7084b450cb24f2fa08ad126a6925e7
Aug 10 23:25:41 dist-test-slave-rmhg krb5kdc[21380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404340, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:41.055446 21718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:41.035752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:53069 (local address 127.14.175.126:33851)
0810 23:25:41.036253 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:41.036260 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:41.037558 (+ 1298us) server_negotiation.cc:407] Connection header received
0810 23:25:41.039495 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:41.039501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:41.039584 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:41.039708 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:41.041521 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:41.042481 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:41.044191 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:41.044666 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:41.044789 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:41.049000 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:41.049006 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:41.049009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:41.049048 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:41.051806 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:41.052873 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:41.052878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:41.052881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:41.053015 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:41.053550 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:41.053554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:41.053555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:41.053848 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:41.053965 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:41.054970 (+ 1005us) server_negotiation.cc:299] Negotiation successful
0810 23:25:41.055176 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":237,"threads_started":1}
WARNING: no policy specified for kudu/127.14.175.66@KRBTEST.COM; defaulting to no policy
I20260810 23:25:41.057049 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:41.036455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:33851 (local address 127.14.175.65:53069)
0810 23:25:41.037375 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:41.037425 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:41.039208 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:41.039907 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:41.039915 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:41.040604 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:41.041345 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:41.041367 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:41.042666 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:41.042670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:41.043992 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:41.044008 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:41.044933 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:41.044941 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:41.045240 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:41.045995 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:41.046035 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:41.048659 (+ 2624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:41.051977 (+ 3318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:41.051989 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:41.052009 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:41.052724 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:41.053126 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:41.053132 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:41.053136 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:41.053439 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:41.053981 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:41.053991 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:41.054522 (+ 531us) client_negotiation.cc:769] Sending connection context
0810 23:25:41.054881 (+ 359us) client_negotiation.cc:240] Negotiation successful
0810 23:25:41.055489 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":358,"threads_started":1}
I20260810 23:25:41.058764 21716 heartbeater.cc:347] Connected to a master server at 127.14.175.126:33851
I20260810 23:25:41.059064 21716 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.66@KRBTEST.COM" created.
I20260810 23:25:41.059895 21716 heartbeater.cc:511] Master 127.14.175.126:33851 requested a full tablet report, sending...
I20260810 23:25:41.062304 21433 ts_manager.cc:194] Registered new tserver with Master: 689c75c2475d4978b0680bdf6c01711b (127.14.175.65:45747)
I20260810 23:25:41.063830 21433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.65@KRBTEST.COM'} at 127.14.175.65:53069
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
I20260810 23:25:41.088747 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.66:0
--local_ip_for_outbound_sockets=127.14.175.66
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:33851
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:44133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:41.441927 21724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:41.442245 21724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:41.442394 21724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:41.450083 21724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:41.450237 21724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:41.450397 21724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.66
I20260810 23:25:41.460772 21724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:44133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:33851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.66
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:41.462472 21724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:41.463852 21724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:41.477774 21730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:41.477936 21731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:41.477983 21733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:41.479367 21724 server_base.cc:1034] running on GCE node
I20260810 23:25:41.480168 21724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:41.481313 21724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:41.482561 21724 hybrid_clock.cc:648] HybridClock initialized: now 1786404341482500 us; error 56 us; skew 500 ppm
I20260810 23:25:41.482939 21724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:41 dist-test-slave-rmhg krb5kdc[21380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404341, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:41.486654 21724 init.cc:377] Logged in from keytab as kudu/127.14.175.66@KRBTEST.COM (short username kudu)
I20260810 23:25:41.487962 21724 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:41.488098 21724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:41.488428 21724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:41.491765 21724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-1/data/instance:
uuid: "e5d67062624f4d4dba780945a17b4f18"
format_stamp: "Formatted at 2026-08-10 23:25:41 on dist-test-slave-rmhg"
server_key: "63f7a391a64f9b17fb5c5949724957dc"
server_key_iv: "339b803f74f71e5fc5fe5cf183539ded"
server_key_version: "encryptionkey@0"
I20260810 23:25:41.492702 21724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance:
uuid: "e5d67062624f4d4dba780945a17b4f18"
format_stamp: "Formatted at 2026-08-10 23:25:41 on dist-test-slave-rmhg"
server_key: "63f7a391a64f9b17fb5c5949724957dc"
server_key_iv: "339b803f74f71e5fc5fe5cf183539ded"
server_key_version: "encryptionkey@0"
I20260810 23:25:41.498667 21724 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260810 23:25:41.502838 21741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:41.504434 21724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260810 23:25:41.504684 21724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-1/wal
uuid: "e5d67062624f4d4dba780945a17b4f18"
format_stamp: "Formatted at 2026-08-10 23:25:41 on dist-test-slave-rmhg"
server_key: "63f7a391a64f9b17fb5c5949724957dc"
server_key_iv: "339b803f74f71e5fc5fe5cf183539ded"
server_key_version: "encryptionkey@0"
I20260810 23:25:41.504931 21724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:41.521235 21724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:41.526109 21724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:41.526417 21724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:41.527478 21724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:25:41.528980 21724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:25:41.529083 21724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:41.529227 21724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:25:41.529299 21724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:41.561425 21724 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.66:43157
I20260810 23:25:41.561448 21905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.66:43157 every 8 connection(s)
I20260810 23:25:41.562456 21724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
I20260810 23:25:41.563581 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 21724
I20260810 23:25:41.563787 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance
I20260810 23:25:41.564373 15037 external_mini_cluster.cc:1428] Setting key 49dd89bb8c65b13dd176736358637df6
Aug 10 23:25:41 dist-test-slave-rmhg krb5kdc[21380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404341, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:41.586726 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:41.565631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:49643 (local address 127.14.175.126:33851)
0810 23:25:41.566160 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:41.566166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:41.567444 (+ 1278us) server_negotiation.cc:407] Connection header received
0810 23:25:41.569625 (+ 2181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:41.569632 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:41.569727 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:41.569843 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:41.571901 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:41.572959 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:41.574926 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:41.575493 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:41.575641 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:41.580328 (+ 4687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:41.580333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:41.580336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:41.580375 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:41.583050 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:41.584066 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:41.584071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:41.584073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:41.584221 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:41.584721 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:41.584725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:41.584727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:41.585041 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:41.585225 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:41.586076 (+ 851us) server_negotiation.cc:299] Negotiation successful
0810 23:25:41.586347 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":249,"threads_started":1}
WARNING: no policy specified for kudu/127.14.175.67@KRBTEST.COM; defaulting to no policy
I20260810 23:25:41.587879 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:41.566345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:33851 (local address 127.14.175.66:49643)
0810 23:25:41.567240 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:41.567303 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:41.569283 (+ 1980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:41.570083 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:41.570093 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:41.570742 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:41.571658 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:41.571681 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:41.573217 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:41.573223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:41.574737 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:41.574753 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:41.575781 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:41.575789 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:41.576115 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:41.576913 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:41.576944 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:41.580037 (+ 3093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:41.583245 (+ 3208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:41.583256 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:41.583273 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:41.583902 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:41.584346 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:41.584351 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:41.584355 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:41.584590 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:41.585222 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:41.585232 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:41.585642 (+ 410us) client_negotiation.cc:769] Sending connection context
0810 23:25:41.585914 (+ 272us) client_negotiation.cc:240] Negotiation successful
0810 23:25:41.586585 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":371,"threads_started":1}
I20260810 23:25:41.589745 21906 heartbeater.cc:347] Connected to a master server at 127.14.175.126:33851
I20260810 23:25:41.590094 21906 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.67@KRBTEST.COM" created.
I20260810 23:25:41.590953 21906 heartbeater.cc:511] Master 127.14.175.126:33851 requested a full tablet report, sending...
I20260810 23:25:41.592877 21433 ts_manager.cc:194] Registered new tserver with Master: e5d67062624f4d4dba780945a17b4f18 (127.14.175.66:43157)
I20260810 23:25:41.594002 21433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.66@KRBTEST.COM'} at 127.14.175.66:49643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
I20260810 23:25:41.620025 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.67:0
--local_ip_for_outbound_sockets=127.14.175.67
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:33851
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:44133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:41.975412 21917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:41.975749 21917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:41.975885 21917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:41.983507 21917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:41.983662 21917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:41.983824 21917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.67
I20260810 23:25:41.994410 21917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:44133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:33851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.67
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:41.995824 21917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:41.997200 21917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:42.010345 21923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:42.010345 21924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:42.010967 21926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:42.011190 21917 server_base.cc:1034] running on GCE node
I20260810 23:25:42.011864 21917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:42.012898 21917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:42.014204 21917 hybrid_clock.cc:648] HybridClock initialized: now 1786404342014125 us; error 59 us; skew 500 ppm
I20260810 23:25:42.014564 21917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:42 dist-test-slave-rmhg krb5kdc[21380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404342, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:42.017940 21917 init.cc:377] Logged in from keytab as kudu/127.14.175.67@KRBTEST.COM (short username kudu)
I20260810 23:25:42.019335 21917 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:42.019445 21917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:42.019757 21917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:42.022826 21917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-2/data/instance:
uuid: "b0cb7c406beb4520840b09312f97d074"
format_stamp: "Formatted at 2026-08-10 23:25:42 on dist-test-slave-rmhg"
server_key: "d73a8ab12f75175711a94472c9a87770"
server_key_iv: "3ba2f3062a876af51508ebff7a604ca0"
server_key_version: "encryptionkey@0"
I20260810 23:25:42.023686 21917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance:
uuid: "b0cb7c406beb4520840b09312f97d074"
format_stamp: "Formatted at 2026-08-10 23:25:42 on dist-test-slave-rmhg"
server_key: "d73a8ab12f75175711a94472c9a87770"
server_key_iv: "3ba2f3062a876af51508ebff7a604ca0"
server_key_version: "encryptionkey@0"
I20260810 23:25:42.029002 21917 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:25:42.032711 21932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:42.034116 21917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:25:42.034260 21917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-2/wal
uuid: "b0cb7c406beb4520840b09312f97d074"
format_stamp: "Formatted at 2026-08-10 23:25:42 on dist-test-slave-rmhg"
server_key: "d73a8ab12f75175711a94472c9a87770"
server_key_iv: "3ba2f3062a876af51508ebff7a604ca0"
server_key_version: "encryptionkey@0"
I20260810 23:25:42.034463 21917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:42.058517 21917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:42.062806 21917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:42.063081 21917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:42.064103 21917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:25:42.065517 21917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:25:42.065609 21917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:42.065719 21917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:25:42.065789 21917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:42.067024 21716 heartbeater.cc:503] Master 127.14.175.126:33851 was elected leader, sending a full tablet report...
I20260810 23:25:42.097518 21917 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.67:43199
I20260810 23:25:42.097582 22113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.67:43199 every 8 connection(s)
I20260810 23:25:42.098495 21917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
I20260810 23:25:42.104902 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 21917
I20260810 23:25:42.105082 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance
I20260810 23:25:42.105557 15037 external_mini_cluster.cc:1428] Setting key fd10a09b055f3d7d3b836e58e3825d5a
Aug 10 23:25:42 dist-test-slave-rmhg krb5kdc[21380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404342, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:42.119469 22117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:42.101077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:47691 (local address 127.14.175.126:33851)
0810 23:25:42.101533 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:42.101537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:42.102471 (+ 934us) server_negotiation.cc:407] Connection header received
0810 23:25:42.104479 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:42.104485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:42.104592 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:42.104694 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:42.106555 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:42.107561 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:42.109451 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:42.109876 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:42.109990 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:42.113497 (+ 3507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:42.113502 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:42.113505 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:42.113549 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:42.116150 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:42.116950 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:42.116954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:42.116956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:42.117089 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:42.117585 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:42.117588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:42.117591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:42.117881 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:42.118026 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:42.118796 (+ 770us) server_negotiation.cc:299] Negotiation successful
0810 23:25:42.119063 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":239,"threads_started":1}
I20260810 23:25:42.120625 22118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:42.101586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:33851 (local address 127.14.175.67:47691)
0810 23:25:42.102291 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:42.102342 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:42.104136 (+ 1794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:42.104958 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:42.104967 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:42.105613 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:42.106351 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:42.106375 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:42.107724 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:42.107728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:42.109268 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:42.109282 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:42.110108 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:42.110117 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:42.110346 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:42.111005 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:42.111029 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:42.113264 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:42.116305 (+ 3041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:42.116316 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:42.116337 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:42.116812 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:42.117247 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:42.117252 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:42.117256 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:42.117457 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:42.118025 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:42.118034 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:42.118403 (+ 369us) client_negotiation.cc:769] Sending connection context
0810 23:25:42.118668 (+ 265us) client_negotiation.cc:240] Negotiation successful
0810 23:25:42.119264 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":278,"threads_started":1}
I20260810 23:25:42.122197 22114 heartbeater.cc:347] Connected to a master server at 127.14.175.126:33851
I20260810 23:25:42.122515 22114 heartbeater.cc:464] Registering TS with master...
I20260810 23:25:42.123206 22114 heartbeater.cc:511] Master 127.14.175.126:33851 requested a full tablet report, sending...
I20260810 23:25:42.125005 21433 ts_manager.cc:194] Registered new tserver with Master: b0cb7c406beb4520840b09312f97d074 (127.14.175.67:43199)
I20260810 23:25:42.125907 21433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.67@KRBTEST.COM'} at 127.14.175.67:47691
I20260810 23:25:42.134378 15037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:25:42.158289 22117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:42.139217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41118 (local address 127.14.175.126:33851)
0810 23:25:42.139425 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:42.139431 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:42.139942 (+ 511us) server_negotiation.cc:407] Connection header received
0810 23:25:42.140090 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:42.140094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:42.140168 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:42.140263 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:42.141828 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:42.142623 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:42.143906 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:42.144350 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:42.144453 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:42.152707 (+ 8254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:42.152714 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:42.152718 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:42.152764 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:42.155426 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:42.156241 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:42.156246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:42.156249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:42.156396 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:42.156823 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:42.156827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:42.156829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:42.157265 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:42.157489 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:42.157820 (+ 331us) server_negotiation.cc:299] Negotiation successful
0810 23:25:42.158042 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 23:25:42.163148 21433 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:25:42.170406 21433 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:25:42.194025 22133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:42.183842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50714 (local address 127.14.175.65:45747)
0810 23:25:42.184482 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:42.184492 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:42.185429 (+ 937us) server_negotiation.cc:407] Connection header received
0810 23:25:42.185572 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:42.185581 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:42.185848 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:42.186067 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:25:42.186879 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:42.188446 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:42.191225 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:42.192591 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:42.192685 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:42.192920 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:42.193505 (+ 585us) server_negotiation.cc:299] Negotiation successful
0810 23:25:42.193693 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":253,"threads_started":1}
I20260810 23:25:42.194041 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:42.183704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:45747 (local address 127.0.0.1:50714)
0810 23:25:42.184931 (+ 1227us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:42.184979 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:42.185219 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:42.186072 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:42.186080 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:42.186124 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:25:42.186636 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:42.186655 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:42.188647 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:42.188652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:42.190995 (+ 2343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:42.191015 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:42.192952 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:42.192957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:42.193218 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:42.193278 (+ 60us) client_negotiation.cc:769] Sending connection context
0810 23:25:42.193430 (+ 152us) client_negotiation.cc:240] Negotiation successful
0810 23:25:42.193652 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"thread_start_us":1033,"threads_started":1}
I20260810 23:25:42.195771 22132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:42.184036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:43199 (local address 127.0.0.1:42086)
0810 23:25:42.185077 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:42.185096 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:42.185274 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:42.187616 (+ 2342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:42.187620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:42.187647 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:25:42.188533 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:42.188544 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:42.190575 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:42.190581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:42.192647 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:42.192668 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:42.194699 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:42.194705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:42.195003 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:42.195082 (+ 79us) client_negotiation.cc:769] Sending connection context
0810 23:25:42.195169 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 23:25:42.195349 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912,"mutex_wait_us":138,"thread_start_us":1047,"threads_started":1}
I20260810 23:25:42.195950 22140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:42.186661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40538 (local address 127.14.175.66:43157)
0810 23:25:42.187260 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:42.187269 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:42.187296 (+ 27us) server_negotiation.cc:407] Connection header received
0810 23:25:42.187389 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:42.187396 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:42.187629 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:42.187807 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:25:42.188657 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:42.189759 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:42.192896 (+ 3137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:42.194411 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:42.194565 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:42.194770 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:42.195468 (+ 698us) server_negotiation.cc:299] Negotiation successful
0810 23:25:42.195636 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":306,"threads_started":1}
I20260810 23:25:42.196336 22138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:42.186175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:43157 (local address 127.0.0.1:40538)
0810 23:25:42.186963 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:42.186985 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:42.187108 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:42.187952 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:42.187956 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:42.187977 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:25:42.188431 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:42.188447 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:42.189911 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:42.189915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:42.192720 (+ 2805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:42.192731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:42.194682 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:42.194690 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:42.195008 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:42.195079 (+ 71us) client_negotiation.cc:769] Sending connection context
0810 23:25:42.195169 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 23:25:42.195349 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"mutex_wait_us":334,"thread_start_us":510,"threads_started":1}
I20260810 23:25:42.196753 22137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:42.184785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42086 (local address 127.14.175.67:43199)
0810 23:25:42.185937 (+ 1152us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:42.185948 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:42.185980 (+ 32us) server_negotiation.cc:407] Connection header received
0810 23:25:42.186073 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:42.186081 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:42.186296 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:42.187290 (+ 994us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:25:42.188764 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:42.190413 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:42.192843 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:42.194197 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:42.194324 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:42.194526 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:42.195255 (+ 729us) server_negotiation.cc:299] Negotiation successful
0810 23:25:42.195434 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":983,"thread_start_us":821,"threads_started":1}
I20260810 23:25:42.199081 21608 tablet_service.cc:1467] Processing CreateTablet for tablet 4dec83a0896a4b0197401fc8830aa1d6 (DEFAULT_TABLE table=test-table [id=123ef55faee14e5289151074277201d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:25:42.200431 21608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dec83a0896a4b0197401fc8830aa1d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:42.200284 21810 tablet_service.cc:1467] Processing CreateTablet for tablet 4dec83a0896a4b0197401fc8830aa1d6 (DEFAULT_TABLE table=test-table [id=123ef55faee14e5289151074277201d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:25:42.200631 22014 tablet_service.cc:1467] Processing CreateTablet for tablet 4dec83a0896a4b0197401fc8830aa1d6 (DEFAULT_TABLE table=test-table [id=123ef55faee14e5289151074277201d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:25:42.201869 21810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dec83a0896a4b0197401fc8830aa1d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:42.201947 22014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dec83a0896a4b0197401fc8830aa1d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:42.213553 22141 tablet_bootstrap.cc:492] T 4dec83a0896a4b0197401fc8830aa1d6 P 689c75c2475d4978b0680bdf6c01711b: Bootstrap starting.
I20260810 23:25:42.217435 22141 tablet_bootstrap.cc:654] T 4dec83a0896a4b0197401fc8830aa1d6 P 689c75c2475d4978b0680bdf6c01711b: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:42.218608 22142 tablet_bootstrap.cc:492] T 4dec83a0896a4b0197401fc8830aa1d6 P e5d67062624f4d4dba780945a17b4f18: Bootstrap starting.
I20260810 23:25:42.219200 22141 log.cc:824] T 4dec83a0896a4b0197401fc8830aa1d6 P 689c75c2475d4978b0680bdf6c01711b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:25:42.222638 22142 tablet_bootstrap.cc:654] T 4dec83a0896a4b0197401fc8830aa1d6 P e5d67062624f4d4dba780945a17b4f18: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:42.222940 22141 tablet_bootstrap.cc:492] T 4dec83a0896a4b0197401fc8830aa1d6 P 689c75c2475d4978b0680bdf6c01711b: No bootstrap required, opened a new log
I20260810 23:25:42.223273 22141 ts_tablet_manager.cc:1397] T 4dec83a0896a4b0197401fc8830aa1d6 P 689c75c2475d4978b0680bdf6c01711b: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260810 23:25:42.224054 22142 log.cc:824] T 4dec83a0896a4b0197401fc8830aa1d6 P e5d67062624f4d4dba780945a17b4f18: Log is configured to *not* fsync() on all Append() calls
I20260810 23:25:42.225116 22143 tablet_bootstrap.cc:492] T 4dec83a0896a4b0197401fc8830aa1d6 P b0cb7c406beb4520840b09312f97d074: Bootstrap starting.
I20260810 23:25:42.227437 22142 tablet_bootstrap.cc:492] T 4dec83a0896a4b0197401fc8830aa1d6 P e5d67062624f4d4dba780945a17b4f18: No bootstrap required, opened a new log
I20260810 23:25:42.227720 22142 ts_tablet_manager.cc:1397] T 4dec83a0896a4b0197401fc8830aa1d6 P e5d67062624f4d4dba780945a17b4f18: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260810 23:25:42.229440 22143 tablet_bootstrap.cc:654] T 4dec83a0896a4b0197401fc8830aa1d6 P b0cb7c406beb4520840b09312f97d074: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:42.230847 22143 log.cc:824] T 4dec83a0896a4b0197401fc8830aa1d6 P b0cb7c406beb4520840b09312f97d074: Log is configured to *not* fsync() on all Append() calls
I20260810 23:25:42.233618 22141 raft_consensus.cc:348] T 4dec83a0896a4b0197401fc8830aa1d6 P 689c75c2475d4978b0680bdf6c01711b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689c75c2475d4978b0680bdf6c01711b" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 45747 } } peers { permanent_uuid: "b0cb7c406beb4520840b09312f97d074" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43199 } } peers { permanent_uuid: "e5d67062624f4d4dba780945a17b4f18" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 43157 } }
I20260810 23:25:42.234129 22141 raft_consensus.cc:374] T 4dec83a0896a4b0197401fc8830aa1d6 P 689c75c2475d4978b0680bdf6c01711b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:42.234248 22141 raft_consensus.cc:729] T 4dec83a0896a4b0197401fc8830aa1d6 P 689c75c2475d4978b0680bdf6c01711b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 689c75c2475d4978b0680bdf6c01711b, State: Initialized, Role: FOLLOWER
I20260810 23:25:42.234575 22143 tablet_bootstrap.cc:492] T 4dec83a0896a4b0197401fc8830aa1d6 P b0cb7c406beb4520840b09312f97d074: No bootstrap required, opened a new log
I20260810 23:25:42.234881 22143 ts_tablet_manager.cc:1397] T 4dec83a0896a4b0197401fc8830aa1d6 P b0cb7c406beb4520840b09312f97d074: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.004s
I20260810 23:25:42.234968 22141 consensus_queue.cc:260] T 4dec83a0896a4b0197401fc8830aa1d6 P 689c75c2475d4978b0680bdf6c01711b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689c75c2475d4978b0680bdf6c01711b" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 45747 } } peers { permanent_uuid: "b0cb7c406beb4520840b09312f97d074" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43199 } } peers { permanent_uuid: "e5d67062624f4d4dba780945a17b4f18" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 43157 } }
I20260810 23:25:42.235532 22142 raft_consensus.cc:348] T 4dec83a0896a4b0197401fc8830aa1d6 P e5d67062624f4d4dba780945a17b4f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689c75c2475d4978b0680bdf6c01711b" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 45747 } } peers { permanent_uuid: "b0cb7c406beb4520840b09312f97d074" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43199 } } peers { permanent_uuid: "e5d67062624f4d4dba780945a17b4f18" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 43157 } }
I20260810 23:25:42.236013 22142 raft_consensus.cc:374] T 4dec83a0896a4b0197401fc8830aa1d6 P e5d67062624f4d4dba780945a17b4f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:42.236122 22142 raft_consensus.cc:729] T 4dec83a0896a4b0197401fc8830aa1d6 P e5d67062624f4d4dba780945a17b4f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5d67062624f4d4dba780945a17b4f18, State: Initialized, Role: FOLLOWER
I20260810 23:25:42.236756 22142 consensus_queue.cc:260] T 4dec83a0896a4b0197401fc8830aa1d6 P e5d67062624f4d4dba780945a17b4f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689c75c2475d4978b0680bdf6c01711b" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 45747 } } peers { permanent_uuid: "b0cb7c406beb4520840b09312f97d074" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43199 } } peers { permanent_uuid: "e5d67062624f4d4dba780945a17b4f18" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 43157 } }
I20260810 23:25:42.237040 22141 ts_tablet_manager.cc:1428] T 4dec83a0896a4b0197401fc8830aa1d6 P 689c75c2475d4978b0680bdf6c01711b: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260810 23:25:42.238045 21906 heartbeater.cc:503] Master 127.14.175.126:33851 was elected leader, sending a full tablet report...
I20260810 23:25:42.238871 22142 ts_tablet_manager.cc:1428] T 4dec83a0896a4b0197401fc8830aa1d6 P e5d67062624f4d4dba780945a17b4f18: Time spent starting tablet: real 0.011s user 0.008s sys 0.005s
I20260810 23:25:42.245682 22143 raft_consensus.cc:348] T 4dec83a0896a4b0197401fc8830aa1d6 P b0cb7c406beb4520840b09312f97d074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689c75c2475d4978b0680bdf6c01711b" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 45747 } } peers { permanent_uuid: "b0cb7c406beb4520840b09312f97d074" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43199 } } peers { permanent_uuid: "e5d67062624f4d4dba780945a17b4f18" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 43157 } }
I20260810 23:25:42.246241 22143 raft_consensus.cc:374] T 4dec83a0896a4b0197401fc8830aa1d6 P b0cb7c406beb4520840b09312f97d074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:42.246369 22143 raft_consensus.cc:729] T 4dec83a0896a4b0197401fc8830aa1d6 P b0cb7c406beb4520840b09312f97d074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0cb7c406beb4520840b09312f97d074, State: Initialized, Role: FOLLOWER
I20260810 23:25:42.246994 22143 consensus_queue.cc:260] T 4dec83a0896a4b0197401fc8830aa1d6 P b0cb7c406beb4520840b09312f97d074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689c75c2475d4978b0680bdf6c01711b" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 45747 } } peers { permanent_uuid: "b0cb7c406beb4520840b09312f97d074" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43199 } } peers { permanent_uuid: "e5d67062624f4d4dba780945a17b4f18" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 43157 } }
I20260810 23:25:42.248117 22114 heartbeater.cc:503] Master 127.14.175.126:33851 was elected leader, sending a full tablet report...
I20260810 23:25:42.248827 22143 ts_tablet_manager.cc:1428] T 4dec83a0896a4b0197401fc8830aa1d6 P b0cb7c406beb4520840b09312f97d074: Time spent starting tablet: real 0.014s user 0.007s sys 0.008s
W20260810 23:25:42.285737 21717 tablet.cc:2369] T 4dec83a0896a4b0197401fc8830aa1d6 P 689c75c2475d4978b0680bdf6c01711b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:25:42.314723 21907 tablet.cc:2369] T 4dec83a0896a4b0197401fc8830aa1d6 P e5d67062624f4d4dba780945a17b4f18: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:25:42.350103 22116 tablet.cc:2369] T 4dec83a0896a4b0197401fc8830aa1d6 P b0cb7c406beb4520840b09312f97d074: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:25:42.502162 22151 raft_consensus.cc:482] T 4dec83a0896a4b0197401fc8830aa1d6 P 689c75c2475d4978b0680bdf6c01711b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:25:42.502477 22151 raft_consensus.cc:504] T 4dec83a0896a4b0197401fc8830aa1d6 P 689c75c2475d4978b0680bdf6c01711b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689c75c2475d4978b0680bdf6c01711b" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 45747 } } peers { permanent_uuid: "b0cb7c406beb4520840b09312f97d074" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43199 } } peers { permanent_uuid: "e5d67062624f4d4dba780945a17b4f18" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 43157 } }
I20260810 23:25:42.504242 22151 leader_election.cc:288] T 4dec83a0896a4b0197401fc8830aa1d6 P 689c75c2475d4978b0680bdf6c01711b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0cb7c406beb4520840b09312f97d074 (127.14.175.67:43199), e5d67062624f4d4dba780945a17b4f18 (127.14.175.66:43157)
I20260810 23:25:42.511673 22137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:42.504814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:57625 (local address 127.14.175.67:43199)
0810 23:25:42.505035 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:42.505042 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:42.505962 (+ 920us) server_negotiation.cc:407] Connection header received
0810 23:25:42.506162 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:42.506169 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:42.506248 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:42.506317 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:25:42.506971 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:42.507737 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:42.509843 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:42.510711 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:42.510866 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:42.510942 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:42.511251 (+ 309us) server_negotiation.cc:299] Negotiation successful
0810 23:25:42.511392 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 23:25:42.511776 22155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:42.504929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:43199 (local address 127.14.175.65:57625)
0810 23:25:42.505841 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:42.505857 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:42.506003 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:42.506364 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:42.506370 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:42.506394 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:25:42.506828 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:42.506842 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:42.507871 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:42.507875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:42.509697 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:42.509709 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:42.510941 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:42.510949 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:42.511087 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:42.511123 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 23:25:42.511199 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 23:25:42.511332 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":41,"thread_start_us":474,"threads_started":1}
I20260810 23:25:42.511878 22140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:42.504888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:40939 (local address 127.14.175.66:43157)
0810 23:25:42.505128 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:42.505134 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:42.506065 (+ 931us) server_negotiation.cc:407] Connection header received
0810 23:25:42.506243 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:42.506247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:42.506311 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:42.506394 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:25:42.507022 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:42.507732 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:42.509841 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:42.510715 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:42.510861 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:42.510940 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:42.511497 (+ 557us) server_negotiation.cc:299] Negotiation successful
0810 23:25:42.511645 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 23:25:42.512387 22158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:42.504969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:43157 (local address 127.14.175.65:40939)
0810 23:25:42.505930 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:42.505950 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:42.506108 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:42.506425 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:42.506430 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:42.506456 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:25:42.506845 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:42.506854 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:42.507869 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:42.507874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:42.509669 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:42.509690 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:42.510994 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:42.511000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:42.511147 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:42.511171 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 23:25:42.511330 (+ 159us) client_negotiation.cc:240] Negotiation successful
0810 23:25:42.511465 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"mutex_wait_us":61,"thread_start_us":452,"threads_started":1}
I20260810 23:25:42.513024 22042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dec83a0896a4b0197401fc8830aa1d6" candidate_uuid: "689c75c2475d4978b0680bdf6c01711b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0cb7c406beb4520840b09312f97d074" is_pre_election: true
I20260810 23:25:42.513350 21835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dec83a0896a4b0197401fc8830aa1d6" candidate_uuid: "689c75c2475d4978b0680bdf6c01711b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5d67062624f4d4dba780945a17b4f18" is_pre_election: true
I20260810 23:25:42.513602 22042 raft_consensus.cc:2454] T 4dec83a0896a4b0197401fc8830aa1d6 P b0cb7c406beb4520840b09312f97d074 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 689c75c2475d4978b0680bdf6c01711b in term 0.
I20260810 23:25:42.513862 21835 raft_consensus.cc:2454] T 4dec83a0896a4b0197401fc8830aa1d6 P e5d67062624f4d4dba780945a17b4f18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 689c75c2475d4978b0680bdf6c01711b in term 0.
I20260810 23:25:42.514308 21546 leader_election.cc:302] T 4dec83a0896a4b0197401fc8830aa1d6 P 689c75c2475d4978b0680bdf6c01711b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 689c75c2475d4978b0680bdf6c01711b, b0cb7c406beb4520840b09312f97d074; no voters:
I20260810 23:25:42.514716 22151 raft_consensus.cc:2781] T 4dec83a0896a4b0197401fc8830aa1d6 P 689c75c2475d4978b0680bdf6c01711b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:25:42.514838 22151 raft_consensus.cc:482] T 4dec83a0896a4b0197401fc8830aa1d6 P 689c75c2475d4978b0680bdf6c01711b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:25:42.514932 22151 raft_consensus.cc:3037] T 4dec83a0896a4b0197401fc8830aa1d6 P 689c75c2475d4978b0680bdf6c01711b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:42.517127 22151 raft_consensus.cc:504] T 4dec83a0896a4b0197401fc8830aa1d6 P 689c75c2475d4978b0680bdf6c01711b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689c75c2475d4978b0680bdf6c01711b" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 45747 } } peers { permanent_uuid: "b0cb7c406beb4520840b09312f97d074" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43199 } } peers { permanent_uuid: "e5d67062624f4d4dba780945a17b4f18" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 43157 } }
I20260810 23:25:42.517822 22151 leader_election.cc:288] T 4dec83a0896a4b0197401fc8830aa1d6 P 689c75c2475d4978b0680bdf6c01711b [CANDIDATE]: Term 1 election: Requested vote from peers b0cb7c406beb4520840b09312f97d074 (127.14.175.67:43199), e5d67062624f4d4dba780945a17b4f18 (127.14.175.66:43157)
I20260810 23:25:42.518342 22042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dec83a0896a4b0197401fc8830aa1d6" candidate_uuid: "689c75c2475d4978b0680bdf6c01711b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0cb7c406beb4520840b09312f97d074"
I20260810 23:25:42.518342 21835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dec83a0896a4b0197401fc8830aa1d6" candidate_uuid: "689c75c2475d4978b0680bdf6c01711b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5d67062624f4d4dba780945a17b4f18"
I20260810 23:25:42.518574 22042 raft_consensus.cc:3037] T 4dec83a0896a4b0197401fc8830aa1d6 P b0cb7c406beb4520840b09312f97d074 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:42.518565 21835 raft_consensus.cc:3037] T 4dec83a0896a4b0197401fc8830aa1d6 P e5d67062624f4d4dba780945a17b4f18 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:42.521837 22042 raft_consensus.cc:2454] T 4dec83a0896a4b0197401fc8830aa1d6 P b0cb7c406beb4520840b09312f97d074 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 689c75c2475d4978b0680bdf6c01711b in term 1.
I20260810 23:25:42.521837 21835 raft_consensus.cc:2454] T 4dec83a0896a4b0197401fc8830aa1d6 P e5d67062624f4d4dba780945a17b4f18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 689c75c2475d4978b0680bdf6c01711b in term 1.
I20260810 23:25:42.522387 21548 leader_election.cc:302] T 4dec83a0896a4b0197401fc8830aa1d6 P 689c75c2475d4978b0680bdf6c01711b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 689c75c2475d4978b0680bdf6c01711b, e5d67062624f4d4dba780945a17b4f18; no voters:
I20260810 23:25:42.522732 22151 raft_consensus.cc:2781] T 4dec83a0896a4b0197401fc8830aa1d6 P 689c75c2475d4978b0680bdf6c01711b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:25:42.523157 22151 raft_consensus.cc:686] T 4dec83a0896a4b0197401fc8830aa1d6 P 689c75c2475d4978b0680bdf6c01711b [term 1 LEADER]: Becoming Leader. State: Replica: 689c75c2475d4978b0680bdf6c01711b, State: Running, Role: LEADER
I20260810 23:25:42.523667 22151 consensus_queue.cc:237] T 4dec83a0896a4b0197401fc8830aa1d6 P 689c75c2475d4978b0680bdf6c01711b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689c75c2475d4978b0680bdf6c01711b" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 45747 } } peers { permanent_uuid: "b0cb7c406beb4520840b09312f97d074" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43199 } } peers { permanent_uuid: "e5d67062624f4d4dba780945a17b4f18" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 43157 } }
I20260810 23:25:42.528898 21432 catalog_manager.cc:5495] T 4dec83a0896a4b0197401fc8830aa1d6 P 689c75c2475d4978b0680bdf6c01711b reported cstate change: term changed from 0 to 1, leader changed from <none> to 689c75c2475d4978b0680bdf6c01711b (127.14.175.65). New cstate: current_term: 1 leader_uuid: "689c75c2475d4978b0680bdf6c01711b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689c75c2475d4978b0680bdf6c01711b" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 45747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0cb7c406beb4520840b09312f97d074" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5d67062624f4d4dba780945a17b4f18" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 43157 } health_report { overall_health: UNKNOWN } } }
I20260810 23:25:42.569296 22133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:42.563333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50728 (local address 127.14.175.65:45747)
0810 23:25:42.563543 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:42.563550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:42.563743 (+ 193us) server_negotiation.cc:407] Connection header received
0810 23:25:42.563993 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:42.563998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:42.564094 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:42.564237 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:25:42.564722 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:42.565505 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:42.566911 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:42.567348 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:42.567465 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:42.568052 (+ 587us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:25:42.568605 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:25:42.568768 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:42.568857 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 23:25:42.568955 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 23:25:42.579198 21835 raft_consensus.cc:1261] T 4dec83a0896a4b0197401fc8830aa1d6 P e5d67062624f4d4dba780945a17b4f18 [term 1 FOLLOWER]: Refusing update from remote peer 689c75c2475d4978b0680bdf6c01711b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:25:42.579198 22042 raft_consensus.cc:1261] T 4dec83a0896a4b0197401fc8830aa1d6 P b0cb7c406beb4520840b09312f97d074 [term 1 FOLLOWER]: Refusing update from remote peer 689c75c2475d4978b0680bdf6c01711b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:25:42.580561 22159 consensus_queue.cc:1035] T 4dec83a0896a4b0197401fc8830aa1d6 P 689c75c2475d4978b0680bdf6c01711b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0cb7c406beb4520840b09312f97d074" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:25:42.580996 22151 consensus_queue.cc:1035] T 4dec83a0896a4b0197401fc8830aa1d6 P 689c75c2475d4978b0680bdf6c01711b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5d67062624f4d4dba780945a17b4f18" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 43157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:25:42.595901 22166 mvcc.cc:205] Tried to move back new op lower bound from 7317112187190562816 to 7317112186980839424. Current Snapshot: MvccSnapshot[applied={T|T < 7317112187190562816}]
I20260810 23:25:42.596463 22163 mvcc.cc:205] Tried to move back new op lower bound from 7317112187190562816 to 7317112186980839424. Current Snapshot: MvccSnapshot[applied={T|T < 7317112187190562816}]
I20260810 23:25:42.608806 21433 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:25:42.609114 21433 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:25:42.613044 21433 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d0627788d1344f2a1f112c190326655: Sending DeleteTablet for 3 replicas of tablet 4dec83a0896a4b0197401fc8830aa1d6
I20260810 23:25:42.614413 21810 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dec83a0896a4b0197401fc8830aa1d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:25:42 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:40538
I20260810 23:25:42.614410 22014 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dec83a0896a4b0197401fc8830aa1d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:25:42 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:42086
I20260810 23:25:42.615216 21608 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dec83a0896a4b0197401fc8830aa1d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:25:42 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:50714
I20260810 23:25:42.615474 22178 tablet_replica.cc:330] T 4dec83a0896a4b0197401fc8830aa1d6 P e5d67062624f4d4dba780945a17b4f18: stopping tablet replica
I20260810 23:25:42.615916 22178 raft_consensus.cc:2229] T 4dec83a0896a4b0197401fc8830aa1d6 P e5d67062624f4d4dba780945a17b4f18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:25:42.616243 22181 tablet_replica.cc:330] T 4dec83a0896a4b0197401fc8830aa1d6 P 689c75c2475d4978b0680bdf6c01711b: stopping tablet replica
I20260810 23:25:42.616479 22178 raft_consensus.cc:2258] T 4dec83a0896a4b0197401fc8830aa1d6 P e5d67062624f4d4dba780945a17b4f18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:25:42.616844 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 21522
I20260810 23:25:42.616797 22181 raft_consensus.cc:2229] T 4dec83a0896a4b0197401fc8830aa1d6 P 689c75c2475d4978b0680bdf6c01711b [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:25:42.617647 22181 raft_consensus.cc:2258] T 4dec83a0896a4b0197401fc8830aa1d6 P 689c75c2475d4978b0680bdf6c01711b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:25:42.620123 22181 ts_tablet_manager.cc:1905] T 4dec83a0896a4b0197401fc8830aa1d6 P 689c75c2475d4978b0680bdf6c01711b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:25:42.626017 22178 ts_tablet_manager.cc:1905] T 4dec83a0896a4b0197401fc8830aa1d6 P e5d67062624f4d4dba780945a17b4f18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:25:42.629050 22181 ts_tablet_manager.cc:1918] T 4dec83a0896a4b0197401fc8830aa1d6 P 689c75c2475d4978b0680bdf6c01711b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:25:42.629434 22181 log.cc:1196] T 4dec83a0896a4b0197401fc8830aa1d6 P 689c75c2475d4978b0680bdf6c01711b: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-0/wal/wals/4dec83a0896a4b0197401fc8830aa1d6
I20260810 23:25:42.630189 22181 ts_tablet_manager.cc:1939] T 4dec83a0896a4b0197401fc8830aa1d6 P 689c75c2475d4978b0680bdf6c01711b: Deleting consensus metadata
I20260810 23:25:42.632509 21419 catalog_manager.cc:4841] TS 689c75c2475d4978b0680bdf6c01711b (127.14.175.65:45747): tablet 4dec83a0896a4b0197401fc8830aa1d6 (table test-table [id=123ef55faee14e5289151074277201d9]) successfully deleted
I20260810 23:25:42.633659 22179 tablet_replica.cc:330] T 4dec83a0896a4b0197401fc8830aa1d6 P b0cb7c406beb4520840b09312f97d074: stopping tablet replica
I20260810 23:25:42.634254 22179 raft_consensus.cc:2229] T 4dec83a0896a4b0197401fc8830aa1d6 P b0cb7c406beb4520840b09312f97d074 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:25:42.634899 22179 raft_consensus.cc:2258] T 4dec83a0896a4b0197401fc8830aa1d6 P b0cb7c406beb4520840b09312f97d074 [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 10 23:25:42 dist-test-slave-rmhg krb5kdc[21380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.65@KRBTEST.COM
I20260810 23:25:42.637028 22179 ts_tablet_manager.cc:1905] T 4dec83a0896a4b0197401fc8830aa1d6 P b0cb7c406beb4520840b09312f97d074: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:25:42.637985 22178 ts_tablet_manager.cc:1918] T 4dec83a0896a4b0197401fc8830aa1d6 P e5d67062624f4d4dba780945a17b4f18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:25:42.638203 22178 log.cc:1196] T 4dec83a0896a4b0197401fc8830aa1d6 P e5d67062624f4d4dba780945a17b4f18: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-1/wal/wals/4dec83a0896a4b0197401fc8830aa1d6
I20260810 23:25:42.638736 22178 ts_tablet_manager.cc:1939] T 4dec83a0896a4b0197401fc8830aa1d6 P e5d67062624f4d4dba780945a17b4f18: Deleting consensus metadata
I20260810 23:25:42.640969 21419 catalog_manager.cc:4841] TS e5d67062624f4d4dba780945a17b4f18 (127.14.175.66:43157): tablet 4dec83a0896a4b0197401fc8830aa1d6 (table test-table [id=123ef55faee14e5289151074277201d9]) successfully deleted
I20260810 23:25:42.644197 22179 ts_tablet_manager.cc:1918] T 4dec83a0896a4b0197401fc8830aa1d6 P b0cb7c406beb4520840b09312f97d074: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:25:42.644517 22179 log.cc:1196] T 4dec83a0896a4b0197401fc8830aa1d6 P b0cb7c406beb4520840b09312f97d074: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404299514076-15037-0/minicluster-data/ts-2/wal/wals/4dec83a0896a4b0197401fc8830aa1d6
I20260810 23:25:42.645002 22179 ts_tablet_manager.cc:1939] T 4dec83a0896a4b0197401fc8830aa1d6 P b0cb7c406beb4520840b09312f97d074: Deleting consensus metadata
I20260810 23:25:42.645884 22133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:42.618082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50740 (local address 127.14.175.65:45747)
0810 23:25:42.618277 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:42.618284 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:42.624204 (+ 5920us) server_negotiation.cc:407] Connection header received
0810 23:25:42.624554 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:42.624560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:42.624648 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:42.624747 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:42.626694 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:42.628020 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:42.629989 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:42.630712 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:42.630816 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:42.637096 (+ 6280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:42.637117 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:42.637138 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:42.637231 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:42.641677 (+ 4446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:42.642689 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:42.642701 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:42.642705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:42.642923 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:42.643448 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:42.643454 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:42.643457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:42.644008 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:42.644363 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:42.645208 (+ 845us) server_negotiation.cc:299] Negotiation successful
0810 23:25:42.645495 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 23:25:42.646649 21703 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:25:42.647023 21418 catalog_manager.cc:4841] TS b0cb7c406beb4520840b09312f97d074 (127.14.175.67:43199): tablet 4dec83a0896a4b0197401fc8830aa1d6 (table test-table [id=123ef55faee14e5289151074277201d9]) successfully deleted
I20260810 23:25:42.746671 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 21522
I20260810 23:25:42.770498 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 21724
Aug 10 23:25:42 dist-test-slave-rmhg krb5kdc[21380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.66@KRBTEST.COM
I20260810 23:25:42.787400 22140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:42.771356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40550 (local address 127.14.175.66:43157)
0810 23:25:42.771548 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:42.771553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:42.771608 (+ 55us) server_negotiation.cc:407] Connection header received
0810 23:25:42.771802 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:42.771806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:42.771869 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:42.771954 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:42.773365 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:42.774149 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:42.775564 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:42.776092 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:42.776213 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:42.780840 (+ 4627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:42.780861 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:42.780879 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:42.780924 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:42.783993 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:42.784916 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:42.784927 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:42.784929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:42.785120 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:42.785757 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:42.785762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:42.785765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:42.786275 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:42.786591 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:42.786906 (+ 315us) server_negotiation.cc:299] Negotiation successful
0810 23:25:42.787111 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 23:25:42.788239 21900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:25:42.888361 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 21724
I20260810 23:25:42.913798 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 21917
Aug 10 23:25:42 dist-test-slave-rmhg krb5kdc[21380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.67@KRBTEST.COM
I20260810 23:25:42.929260 22137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:42.914587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42092 (local address 127.14.175.67:43199)
0810 23:25:42.914774 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:42.914780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:42.914802 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:25:42.914994 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:42.915000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:42.915066 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:42.915171 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:42.916479 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:42.917270 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:42.918542 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:42.918966 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:42.919117 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:42.923661 (+ 4544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:42.923680 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:42.923699 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:42.923745 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:42.926407 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:42.927120 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:42.927129 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:42.927131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:42.927291 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:42.927668 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:42.927672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:42.927674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:42.928061 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:42.928389 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:42.928654 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 23:25:42.928876 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 23:25:42.929916 22109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:25:43.018395 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 21917
I20260810 23:25:43.044152 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 21397
I20260810 23:25:43.045468 21479 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:25:43.156188 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 21397
2026-08-10T23:25:43Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4763 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T23:25:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:25:43Z Disabled control of system clock
I20260810 23:25:43.243362 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:38057
--webserver_interface=127.14.175.126
--webserver_port=0
--builtin_ntp_servers=127.14.175.84:44435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:38057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 23:25:43.598124 22197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:43.598435 22197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:43.598539 22197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:43.605858 22197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:25:43.605988 22197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:43.606026 22197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:43.606084 22197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:25:43.606128 22197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:25:43.615723 22197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:44435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:38057
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:38057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:43.617166 22197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:43.619076 22197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:43.631521 22209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:43.631659 22203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:43.631626 22204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:43.632661 22197 server_base.cc:1034] running on GCE node
I20260810 23:25:43.633558 22197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:43.635016 22197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:43.636233 22197 hybrid_clock.cc:648] HybridClock initialized: now 1786404343636177 us; error 46 us; skew 500 ppm
I20260810 23:25:43.636605 22197 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:25:43.639035 22197 webserver.cc:460] Webserver started at http://127.14.175.126:34357/ using document root <none> and password file <none>
I20260810 23:25:43.639773 22197 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:43.639878 22197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:43.640218 22197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:43.643404 22197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "93cf75a16e734077b891660c9ee175a5"
format_stamp: "Formatted at 2026-08-10 23:25:43 on dist-test-slave-rmhg"
server_key: "3cfbe6bbe8e803f0bc2cec78303f9d74"
server_key_iv: "11b5921126d9627d539c0cad38e7ab3a"
server_key_version: "encryptionkey@0"
I20260810 23:25:43.644266 22197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "93cf75a16e734077b891660c9ee175a5"
format_stamp: "Formatted at 2026-08-10 23:25:43 on dist-test-slave-rmhg"
server_key: "3cfbe6bbe8e803f0bc2cec78303f9d74"
server_key_iv: "11b5921126d9627d539c0cad38e7ab3a"
server_key_version: "encryptionkey@0"
I20260810 23:25:43.650043 22197 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 23:25:43.653955 22216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:43.655488 22197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:25:43.655675 22197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "93cf75a16e734077b891660c9ee175a5"
format_stamp: "Formatted at 2026-08-10 23:25:43 on dist-test-slave-rmhg"
server_key: "3cfbe6bbe8e803f0bc2cec78303f9d74"
server_key_iv: "11b5921126d9627d539c0cad38e7ab3a"
server_key_version: "encryptionkey@0"
I20260810 23:25:43.655926 22197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:43.662410 22197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:43.663372 22197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:43.663682 22197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:43.682408 22197 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:38057
I20260810 23:25:43.682446 22293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:38057 every 8 connection(s)
I20260810 23:25:43.683812 22197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:25:43.687855 22294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:43.688421 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 22197
I20260810 23:25:43.688654 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:25:43.689664 15037 external_mini_cluster.cc:1428] Setting key 16d1cc91c2c229da9606c6521a15b75e
I20260810 23:25:43.698974 22298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:43.692061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54834 (local address 127.14.175.126:38057)
0810 23:25:43.693312 (+ 1251us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:43.693325 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:43.693380 (+ 55us) server_negotiation.cc:407] Connection header received
0810 23:25:43.695147 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:43.695174 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:43.695673 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:43.696104 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:43.697269 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:43.697275 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:43.697289 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:25:43.697305 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:43.697371 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:25:43.697388 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:43.697605 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:43.697742 (+ 137us) server_negotiation.cc:299] Negotiation successful
0810 23:25:43.698009 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":445,"threads_started":1}
I20260810 23:25:43.702720 22294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93cf75a16e734077b891660c9ee175a5: Bootstrap starting.
I20260810 23:25:43.706072 22294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93cf75a16e734077b891660c9ee175a5: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:43.707245 22294 log.cc:824] T 00000000000000000000000000000000 P 93cf75a16e734077b891660c9ee175a5: Log is configured to *not* fsync() on all Append() calls
I20260810 23:25:43.710539 22294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93cf75a16e734077b891660c9ee175a5: No bootstrap required, opened a new log
I20260810 23:25:43.718214 22294 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93cf75a16e734077b891660c9ee175a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93cf75a16e734077b891660c9ee175a5" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38057 } }
I20260810 23:25:43.718621 22294 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93cf75a16e734077b891660c9ee175a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:43.718716 22294 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93cf75a16e734077b891660c9ee175a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93cf75a16e734077b891660c9ee175a5, State: Initialized, Role: FOLLOWER
I20260810 23:25:43.719295 22294 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93cf75a16e734077b891660c9ee175a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93cf75a16e734077b891660c9ee175a5" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38057 } }
I20260810 23:25:43.719529 22294 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93cf75a16e734077b891660c9ee175a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:25:43.719660 22294 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93cf75a16e734077b891660c9ee175a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:25:43.719818 22294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93cf75a16e734077b891660c9ee175a5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:43.722190 22294 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93cf75a16e734077b891660c9ee175a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93cf75a16e734077b891660c9ee175a5" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38057 } }
I20260810 23:25:43.722733 22294 leader_election.cc:302] T 00000000000000000000000000000000 P 93cf75a16e734077b891660c9ee175a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93cf75a16e734077b891660c9ee175a5; no voters:
I20260810 23:25:43.723351 22294 leader_election.cc:288] T 00000000000000000000000000000000 P 93cf75a16e734077b891660c9ee175a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:25:43.723481 22300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93cf75a16e734077b891660c9ee175a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:25:43.724174 22300 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93cf75a16e734077b891660c9ee175a5 [term 1 LEADER]: Becoming Leader. State: Replica: 93cf75a16e734077b891660c9ee175a5, State: Running, Role: LEADER
I20260810 23:25:43.724776 22300 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93cf75a16e734077b891660c9ee175a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93cf75a16e734077b891660c9ee175a5" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38057 } }
I20260810 23:25:43.725637 22294 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93cf75a16e734077b891660c9ee175a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:25:43.728156 22301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93cf75a16e734077b891660c9ee175a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93cf75a16e734077b891660c9ee175a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93cf75a16e734077b891660c9ee175a5" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38057 } } }
I20260810 23:25:43.728164 22302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93cf75a16e734077b891660c9ee175a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93cf75a16e734077b891660c9ee175a5. Latest consensus state: current_term: 1 leader_uuid: "93cf75a16e734077b891660c9ee175a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93cf75a16e734077b891660c9ee175a5" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38057 } } }
I20260810 23:25:43.728768 22302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93cf75a16e734077b891660c9ee175a5 [sys.catalog]: This master's current role is: LEADER
I20260810 23:25:43.728782 22301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93cf75a16e734077b891660c9ee175a5 [sys.catalog]: This master's current role is: LEADER
I20260810 23:25:43.734807 22306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:25:43.740350 22306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:25:43.750480 22306 catalog_manager.cc:1347] Generated new cluster ID: fc7581a26c2440b38db43f1c6a49eb62
I20260810 23:25:43.750638 22306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:25:43.772852 22306 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:25:43.773386 22306 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:25:43.780480 22306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93cf75a16e734077b891660c9ee175a5: Generated new TSK 0
I20260810 23:25:43.781428 22306 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:25:43.788100 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.65:0
--local_ip_for_outbound_sockets=127.14.175.65
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:38057
--builtin_ntp_servers=127.14.175.84:44435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 23:25:44.135219 22324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:44.135566 22324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:44.135692 22324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:44.143201 22324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:44.143321 22324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:44.143471 22324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.65
I20260810 23:25:44.152876 22324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:44435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:38057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.65
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:44.154314 22324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:44.156229 22324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:44.171653 22334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:44.172118 22330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:44.172003 22332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:44.172806 22324 server_base.cc:1034] running on GCE node
I20260810 23:25:44.173548 22324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:44.174674 22324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:44.175899 22324 hybrid_clock.cc:648] HybridClock initialized: now 1786404344175842 us; error 53 us; skew 500 ppm
I20260810 23:25:44.176219 22324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:25:44.178720 22324 webserver.cc:460] Webserver started at http://127.14.175.65:35173/ using document root <none> and password file <none>
I20260810 23:25:44.179428 22324 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:44.179523 22324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:44.179826 22324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:44.183075 22324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-0/data/instance:
uuid: "3dc3795c0472482f8ada6e12961213ec"
format_stamp: "Formatted at 2026-08-10 23:25:44 on dist-test-slave-rmhg"
server_key: "490e485aa1afaf679040b6b8eadf6850"
server_key_iv: "31da64da7dc040c6ed4857a517dea4ea"
server_key_version: "encryptionkey@0"
I20260810 23:25:44.183954 22324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance:
uuid: "3dc3795c0472482f8ada6e12961213ec"
format_stamp: "Formatted at 2026-08-10 23:25:44 on dist-test-slave-rmhg"
server_key: "490e485aa1afaf679040b6b8eadf6850"
server_key_iv: "31da64da7dc040c6ed4857a517dea4ea"
server_key_version: "encryptionkey@0"
I20260810 23:25:44.189851 22324 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260810 23:25:44.193804 22343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:44.195293 22324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260810 23:25:44.195470 22324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-0/wal
uuid: "3dc3795c0472482f8ada6e12961213ec"
format_stamp: "Formatted at 2026-08-10 23:25:44 on dist-test-slave-rmhg"
server_key: "490e485aa1afaf679040b6b8eadf6850"
server_key_iv: "31da64da7dc040c6ed4857a517dea4ea"
server_key_version: "encryptionkey@0"
I20260810 23:25:44.195721 22324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:44.202664 22324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:44.203611 22324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:44.203890 22324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:44.204941 22324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:25:44.206580 22324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:25:44.206686 22324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:44.206802 22324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:25:44.206873 22324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:44.238796 22324 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.65:42033
I20260810 23:25:44.238848 22509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.65:42033 every 8 connection(s)
I20260810 23:25:44.240232 22324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
I20260810 23:25:44.243767 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 22324
I20260810 23:25:44.244016 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance
I20260810 23:25:44.244896 15037 external_mini_cluster.cc:1428] Setting key 632462708b85854dba6a9c92c0f5427a
I20260810 23:25:44.248265 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.66:0
--local_ip_for_outbound_sockets=127.14.175.66
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:38057
--builtin_ntp_servers=127.14.175.84:44435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 23:25:44.249837 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:44.243023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:54423 (local address 127.14.175.126:38057)
0810 23:25:44.243478 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:44.243483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:44.245018 (+ 1535us) server_negotiation.cc:407] Connection header received
0810 23:25:44.247283 (+ 2265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:44.247288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:44.247353 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:44.247484 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:44.249007 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:44.249014 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:44.249019 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:25:44.249043 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:44.249101 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:25:44.249118 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:44.249287 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:44.249456 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 23:25:44.249528 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":252,"threads_started":1}
I20260810 23:25:44.250811 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:44.243750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:38057 (local address 127.14.175.65:54423)
0810 23:25:44.244854 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:44.244897 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:44.246920 (+ 2023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:44.247694 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:44.247703 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:44.247814 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:44.248645 (+ 831us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:25:44.248647 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:44.248692 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:25:44.248794 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:44.249251 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:44.249261 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:44.249268 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 23:25:44.249521 (+ 253us) client_negotiation.cc:240] Negotiation successful
0810 23:25:44.249775 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":363,"threads_started":1}
I20260810 23:25:44.252533 22510 heartbeater.cc:347] Connected to a master server at 127.14.175.126:38057
I20260810 23:25:44.252907 22510 heartbeater.cc:464] Registering TS with master...
I20260810 23:25:44.253739 22510 heartbeater.cc:511] Master 127.14.175.126:38057 requested a full tablet report, sending...
I20260810 23:25:44.256150 22240 ts_manager.cc:194] Registered new tserver with Master: 3dc3795c0472482f8ada6e12961213ec (127.14.175.65:42033)
W20260810 23:25:44.597882 22519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:44.598207 22519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:44.598299 22519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:44.606048 22519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:44.606173 22519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:44.606290 22519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.66
I20260810 23:25:44.615871 22519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:44435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:38057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.66
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:44.617302 22519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:44.619182 22519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:44.633128 22528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:44.633085 22531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:44.633186 22529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:44.634474 22519 server_base.cc:1034] running on GCE node
I20260810 23:25:44.635241 22519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:44.636266 22519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:44.637480 22519 hybrid_clock.cc:648] HybridClock initialized: now 1786404344637414 us; error 58 us; skew 500 ppm
I20260810 23:25:44.637856 22519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:25:44.640307 22519 webserver.cc:460] Webserver started at http://127.14.175.66:45521/ using document root <none> and password file <none>
I20260810 23:25:44.641059 22519 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:44.641188 22519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:44.641566 22519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:44.644767 22519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-1/data/instance:
uuid: "3e17a3a82bb24b8fa3e014c9c70c5d91"
format_stamp: "Formatted at 2026-08-10 23:25:44 on dist-test-slave-rmhg"
server_key: "a86055f29d7299e1968f6ff5b46e3df7"
server_key_iv: "c54cd6cebb30ea7b16e5326d2dbecb06"
server_key_version: "encryptionkey@0"
I20260810 23:25:44.645783 22519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance:
uuid: "3e17a3a82bb24b8fa3e014c9c70c5d91"
format_stamp: "Formatted at 2026-08-10 23:25:44 on dist-test-slave-rmhg"
server_key: "a86055f29d7299e1968f6ff5b46e3df7"
server_key_iv: "c54cd6cebb30ea7b16e5326d2dbecb06"
server_key_version: "encryptionkey@0"
I20260810 23:25:44.651885 22519 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 23:25:44.655884 22541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:44.657491 22519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:25:44.657709 22519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-1/wal
uuid: "3e17a3a82bb24b8fa3e014c9c70c5d91"
format_stamp: "Formatted at 2026-08-10 23:25:44 on dist-test-slave-rmhg"
server_key: "a86055f29d7299e1968f6ff5b46e3df7"
server_key_iv: "c54cd6cebb30ea7b16e5326d2dbecb06"
server_key_version: "encryptionkey@0"
I20260810 23:25:44.657941 22519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:44.664250 22519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:44.665237 22519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:44.665530 22519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:44.666555 22519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:25:44.668286 22519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:25:44.668398 22519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:44.668519 22519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:25:44.668637 22519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:44.705143 22519 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.66:35807
I20260810 23:25:44.705233 22715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.66:35807 every 8 connection(s)
I20260810 23:25:44.706586 22519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
I20260810 23:25:44.714231 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 22519
I20260810 23:25:44.714412 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance
I20260810 23:25:44.714866 15037 external_mini_cluster.cc:1428] Setting key 824a7fd8b758b3cbbca545df9e4417dd
I20260810 23:25:44.714905 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:44.709272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:47783 (local address 127.14.175.126:38057)
0810 23:25:44.709510 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:44.709517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:44.710894 (+ 1377us) server_negotiation.cc:407] Connection header received
0810 23:25:44.712595 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:44.712600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:44.712693 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:44.712788 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:44.714125 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:44.714131 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:44.714135 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:25:44.714154 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:44.714187 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:25:44.714199 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:44.714307 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:44.714557 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 23:25:44.714631 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 23:25:44.716077 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:44.709760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:38057 (local address 127.14.175.66:47783)
0810 23:25:44.710749 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:44.710786 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:44.712318 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:44.713036 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:44.713044 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:44.713141 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:44.713804 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:25:44.713806 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:44.713846 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:25:44.713934 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:44.714289 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:44.714301 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:44.714310 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 23:25:44.714645 (+ 335us) client_negotiation.cc:240] Negotiation successful
0810 23:25:44.714915 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":281,"threads_started":1}
I20260810 23:25:44.717576 22717 heartbeater.cc:347] Connected to a master server at 127.14.175.126:38057
I20260810 23:25:44.717913 22717 heartbeater.cc:464] Registering TS with master...
I20260810 23:25:44.718145 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.67:0
--local_ip_for_outbound_sockets=127.14.175.67
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:38057
--builtin_ntp_servers=127.14.175.84:44435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 23:25:44.718545 22717 heartbeater.cc:511] Master 127.14.175.126:38057 requested a full tablet report, sending...
I20260810 23:25:44.720059 22240 ts_manager.cc:194] Registered new tserver with Master: 3e17a3a82bb24b8fa3e014c9c70c5d91 (127.14.175.66:35807)
W20260810 23:25:45.066319 22721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:45.066641 22721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:45.066735 22721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:45.074288 22721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:45.074414 22721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:45.074527 22721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.67
I20260810 23:25:45.083992 22721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:44435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:38057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.67
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:45.085371 22721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:45.087208 22721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:45.101186 22727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:45.101006 22728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:45.100941 22732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:45.102308 22721 server_base.cc:1034] running on GCE node
I20260810 23:25:45.102950 22721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:45.103948 22721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:45.105217 22721 hybrid_clock.cc:648] HybridClock initialized: now 1786404345105137 us; error 56 us; skew 500 ppm
I20260810 23:25:45.105587 22721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:25:45.107995 22721 webserver.cc:460] Webserver started at http://127.14.175.67:45501/ using document root <none> and password file <none>
I20260810 23:25:45.108767 22721 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:45.108876 22721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:45.109282 22721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:45.112377 22721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-2/data/instance:
uuid: "a96dc7f09d9b49069d1d11c24d5b3548"
format_stamp: "Formatted at 2026-08-10 23:25:45 on dist-test-slave-rmhg"
server_key: "4f1aed065b6a8a6a2f715dbec0e0d2a1"
server_key_iv: "b0276bdecc181459e32c89e3a0651a87"
server_key_version: "encryptionkey@0"
I20260810 23:25:45.113265 22721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance:
uuid: "a96dc7f09d9b49069d1d11c24d5b3548"
format_stamp: "Formatted at 2026-08-10 23:25:45 on dist-test-slave-rmhg"
server_key: "4f1aed065b6a8a6a2f715dbec0e0d2a1"
server_key_iv: "b0276bdecc181459e32c89e3a0651a87"
server_key_version: "encryptionkey@0"
I20260810 23:25:45.118965 22721 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 23:25:45.123029 22738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:45.124781 22721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:25:45.124994 22721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-2/wal
uuid: "a96dc7f09d9b49069d1d11c24d5b3548"
format_stamp: "Formatted at 2026-08-10 23:25:45 on dist-test-slave-rmhg"
server_key: "4f1aed065b6a8a6a2f715dbec0e0d2a1"
server_key_iv: "b0276bdecc181459e32c89e3a0651a87"
server_key_version: "encryptionkey@0"
I20260810 23:25:45.125295 22721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:45.131914 22721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:45.132860 22721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:45.133178 22721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:45.134263 22721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:25:45.135874 22721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:25:45.135972 22721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:45.136101 22721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:25:45.136179 22721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:45.168279 22721 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.67:34707
I20260810 23:25:45.168334 22923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.67:34707 every 8 connection(s)
I20260810 23:25:45.169832 22721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
I20260810 23:25:45.173316 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 22721
I20260810 23:25:45.173518 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance
I20260810 23:25:45.174120 15037 external_mini_cluster.cc:1428] Setting key 6530c72c7140a040055b7794eacaf88b
I20260810 23:25:45.180385 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:45.172731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:49133 (local address 127.14.175.126:38057)
0810 23:25:45.172927 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:45.172933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:45.175881 (+ 2948us) server_negotiation.cc:407] Connection header received
0810 23:25:45.177987 (+ 2106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:45.178000 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:45.178073 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:45.178164 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:45.179614 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:45.179620 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:45.179625 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:25:45.179648 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:45.179687 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:25:45.179699 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:45.179807 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:45.180018 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 23:25:45.180095 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 23:25:45.181550 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:45.173396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:38057 (local address 127.14.175.67:49133)
0810 23:25:45.175672 (+ 2276us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:45.175732 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:45.177690 (+ 1958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:45.178445 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:45.178454 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:45.178576 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:45.179245 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:25:45.179247 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:45.179293 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:25:45.179396 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:45.179790 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:45.179802 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:45.179810 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 23:25:45.180098 (+ 288us) client_negotiation.cc:240] Negotiation successful
0810 23:25:45.180417 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1676,"thread_start_us":1441,"threads_started":1}
I20260810 23:25:45.183202 22925 heartbeater.cc:347] Connected to a master server at 127.14.175.126:38057
I20260810 23:25:45.183588 22925 heartbeater.cc:464] Registering TS with master...
I20260810 23:25:45.184280 22925 heartbeater.cc:511] Master 127.14.175.126:38057 requested a full tablet report, sending...
I20260810 23:25:45.186055 22240 ts_manager.cc:194] Registered new tserver with Master: a96dc7f09d9b49069d1d11c24d5b3548 (127.14.175.67:34707)
I20260810 23:25:45.190639 15037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:25:45.198065 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:45.195336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54850 (local address 127.14.175.126:38057)
0810 23:25:45.195510 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:45.195514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:45.196072 (+ 558us) server_negotiation.cc:407] Connection header received
0810 23:25:45.196312 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:45.196317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:45.196380 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:45.196474 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:45.197466 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:45.197471 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:45.197474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:25:45.197491 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:45.197528 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:25:45.197549 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:45.197679 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:45.197795 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 23:25:45.197848 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 23:25:45.202451 22240 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:25:45.209697 22240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:25:45.253626 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:45.223194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:34707 (local address 127.0.0.1:45302)
0810 23:25:45.247972 (+ 24778us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:45.248028 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:45.248258 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:45.250837 (+ 2579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:45.250842 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:45.250865 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:45.251683 (+ 818us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:25:45.251686 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:45.251703 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:25:45.251752 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:45.252967 (+ 1215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:45.252973 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:45.252981 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 23:25:45.253118 (+ 137us) client_negotiation.cc:240] Negotiation successful
0810 23:25:45.253193 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24468,"thread_start_us":27025,"threads_started":1}
I20260810 23:25:45.253629 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:45.223660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45302 (local address 127.14.175.67:34707)
0810 23:25:45.250246 (+ 26586us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:45.250255 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:45.250297 (+ 42us) server_negotiation.cc:407] Connection header received
0810 23:25:45.250390 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:45.250398 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:45.250628 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:45.251376 (+ 748us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:45.252748 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:45.252756 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:45.252769 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:25:45.252786 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:45.252839 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:25:45.252858 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:45.253034 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:45.253099 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 23:25:45.253250 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26395,"thread_start_us":26214,"threads_started":1}
I20260810 23:25:45.254148 22951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:45.223194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:42033 (local address 127.0.0.1:57358)
0810 23:25:45.248801 (+ 25607us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:45.248818 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:45.248909 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:45.250487 (+ 1578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:45.250495 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:45.250610 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:45.251579 (+ 969us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:25:45.251582 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:45.251620 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:25:45.251710 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:45.252800 (+ 1090us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:45.252815 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:45.252823 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 23:25:45.253023 (+ 200us) client_negotiation.cc:240] Negotiation successful
0810 23:25:45.253136 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25469,"mutex_wait_us":241,"thread_start_us":25077,"threads_started":1}
I20260810 23:25:45.254307 22949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:45.223397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57358 (local address 127.14.175.65:42033)
0810 23:25:45.249778 (+ 26381us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:45.249789 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:45.249826 (+ 37us) server_negotiation.cc:407] Connection header received
0810 23:25:45.249911 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:45.249920 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:45.250235 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:45.251242 (+ 1007us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:45.252594 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:45.252601 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:45.252615 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:25:45.252627 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:45.252668 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:25:45.252682 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:45.252843 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:45.253374 (+ 531us) server_negotiation.cc:299] Negotiation successful
0810 23:25:45.253476 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26189,"thread_start_us":26016,"threads_started":1}
I20260810 23:25:45.255414 22956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:45.224085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49922 (local address 127.14.175.66:35807)
0810 23:25:45.247083 (+ 22998us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:45.247095 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:45.250681 (+ 3586us) server_negotiation.cc:407] Connection header received
0810 23:25:45.250958 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:45.250967 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:45.251695 (+ 728us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:45.251826 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:45.254456 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:45.254465 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:45.254481 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:25:45.254517 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:45.254585 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:25:45.254609 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:45.254734 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:45.254995 (+ 261us) server_negotiation.cc:299] Negotiation successful
0810 23:25:45.255096 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22774,"thread_start_us":23017,"threads_started":1}
I20260810 23:25:45.257725 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:45.223542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:35807 (local address 127.0.0.1:49922)
0810 23:25:45.250520 (+ 26978us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:45.250539 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:45.250795 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:45.252860 (+ 2065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:45.252865 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:45.252887 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:45.254196 (+ 1309us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:25:45.254198 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:45.254219 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:25:45.254271 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:45.254829 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:45.254833 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:45.254837 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:25:45.257350 (+ 2513us) client_negotiation.cc:240] Negotiation successful
0810 23:25:45.257402 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26853,"thread_start_us":25435,"threads_started":1}
I20260810 23:25:45.258740 22412 tablet_service.cc:1467] Processing CreateTablet for tablet 442b0cc01faa4855b76655a33f41a874 (DEFAULT_TABLE table=test-table [id=a94425ff50a345c8a4c2279f5c8bbdbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:25:45.259394 22810 tablet_service.cc:1467] Processing CreateTablet for tablet 442b0cc01faa4855b76655a33f41a874 (DEFAULT_TABLE table=test-table [id=a94425ff50a345c8a4c2279f5c8bbdbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:25:45.260490 22412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442b0cc01faa4855b76655a33f41a874. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:45.260790 22810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442b0cc01faa4855b76655a33f41a874. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:45.262387 22510 heartbeater.cc:503] Master 127.14.175.126:38057 was elected leader, sending a full tablet report...
I20260810 23:25:45.261674 22615 tablet_service.cc:1467] Processing CreateTablet for tablet 442b0cc01faa4855b76655a33f41a874 (DEFAULT_TABLE table=test-table [id=a94425ff50a345c8a4c2279f5c8bbdbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:25:45.263753 22615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442b0cc01faa4855b76655a33f41a874. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:45.274593 22958 tablet_bootstrap.cc:492] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548: Bootstrap starting.
I20260810 23:25:45.276840 22959 tablet_bootstrap.cc:492] T 442b0cc01faa4855b76655a33f41a874 P 3dc3795c0472482f8ada6e12961213ec: Bootstrap starting.
I20260810 23:25:45.279242 22958 tablet_bootstrap.cc:654] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:45.279250 22960 tablet_bootstrap.cc:492] T 442b0cc01faa4855b76655a33f41a874 P 3e17a3a82bb24b8fa3e014c9c70c5d91: Bootstrap starting.
I20260810 23:25:45.280872 22958 log.cc:824] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548: Log is configured to *not* fsync() on all Append() calls
I20260810 23:25:45.281746 22959 tablet_bootstrap.cc:654] T 442b0cc01faa4855b76655a33f41a874 P 3dc3795c0472482f8ada6e12961213ec: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:45.282583 22960 tablet_bootstrap.cc:654] T 442b0cc01faa4855b76655a33f41a874 P 3e17a3a82bb24b8fa3e014c9c70c5d91: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:45.282965 22959 log.cc:824] T 442b0cc01faa4855b76655a33f41a874 P 3dc3795c0472482f8ada6e12961213ec: Log is configured to *not* fsync() on all Append() calls
I20260810 23:25:45.283643 22960 log.cc:824] T 442b0cc01faa4855b76655a33f41a874 P 3e17a3a82bb24b8fa3e014c9c70c5d91: Log is configured to *not* fsync() on all Append() calls
I20260810 23:25:45.284538 22958 tablet_bootstrap.cc:492] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548: No bootstrap required, opened a new log
I20260810 23:25:45.284880 22958 ts_tablet_manager.cc:1397] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260810 23:25:45.286168 22959 tablet_bootstrap.cc:492] T 442b0cc01faa4855b76655a33f41a874 P 3dc3795c0472482f8ada6e12961213ec: No bootstrap required, opened a new log
I20260810 23:25:45.286468 22959 ts_tablet_manager.cc:1397] T 442b0cc01faa4855b76655a33f41a874 P 3dc3795c0472482f8ada6e12961213ec: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260810 23:25:45.286906 22960 tablet_bootstrap.cc:492] T 442b0cc01faa4855b76655a33f41a874 P 3e17a3a82bb24b8fa3e014c9c70c5d91: No bootstrap required, opened a new log
I20260810 23:25:45.287197 22960 ts_tablet_manager.cc:1397] T 442b0cc01faa4855b76655a33f41a874 P 3e17a3a82bb24b8fa3e014c9c70c5d91: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260810 23:25:45.295025 22959 raft_consensus.cc:348] T 442b0cc01faa4855b76655a33f41a874 P 3dc3795c0472482f8ada6e12961213ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc3795c0472482f8ada6e12961213ec" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 42033 } } peers { permanent_uuid: "a96dc7f09d9b49069d1d11c24d5b3548" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 34707 } } peers { permanent_uuid: "3e17a3a82bb24b8fa3e014c9c70c5d91" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 35807 } }
I20260810 23:25:45.295588 22959 raft_consensus.cc:374] T 442b0cc01faa4855b76655a33f41a874 P 3dc3795c0472482f8ada6e12961213ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:45.295708 22959 raft_consensus.cc:729] T 442b0cc01faa4855b76655a33f41a874 P 3dc3795c0472482f8ada6e12961213ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dc3795c0472482f8ada6e12961213ec, State: Initialized, Role: FOLLOWER
I20260810 23:25:45.295637 22958 raft_consensus.cc:348] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc3795c0472482f8ada6e12961213ec" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 42033 } } peers { permanent_uuid: "a96dc7f09d9b49069d1d11c24d5b3548" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 34707 } } peers { permanent_uuid: "3e17a3a82bb24b8fa3e014c9c70c5d91" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 35807 } }
I20260810 23:25:45.296103 22958 raft_consensus.cc:374] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:45.296200 22958 raft_consensus.cc:729] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a96dc7f09d9b49069d1d11c24d5b3548, State: Initialized, Role: FOLLOWER
I20260810 23:25:45.296541 22959 consensus_queue.cc:260] T 442b0cc01faa4855b76655a33f41a874 P 3dc3795c0472482f8ada6e12961213ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc3795c0472482f8ada6e12961213ec" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 42033 } } peers { permanent_uuid: "a96dc7f09d9b49069d1d11c24d5b3548" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 34707 } } peers { permanent_uuid: "3e17a3a82bb24b8fa3e014c9c70c5d91" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 35807 } }
I20260810 23:25:45.296954 22958 consensus_queue.cc:260] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc3795c0472482f8ada6e12961213ec" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 42033 } } peers { permanent_uuid: "a96dc7f09d9b49069d1d11c24d5b3548" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 34707 } } peers { permanent_uuid: "3e17a3a82bb24b8fa3e014c9c70c5d91" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 35807 } }
I20260810 23:25:45.297729 22960 raft_consensus.cc:348] T 442b0cc01faa4855b76655a33f41a874 P 3e17a3a82bb24b8fa3e014c9c70c5d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc3795c0472482f8ada6e12961213ec" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 42033 } } peers { permanent_uuid: "a96dc7f09d9b49069d1d11c24d5b3548" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 34707 } } peers { permanent_uuid: "3e17a3a82bb24b8fa3e014c9c70c5d91" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 35807 } }
I20260810 23:25:45.298233 22960 raft_consensus.cc:374] T 442b0cc01faa4855b76655a33f41a874 P 3e17a3a82bb24b8fa3e014c9c70c5d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:45.298336 22960 raft_consensus.cc:729] T 442b0cc01faa4855b76655a33f41a874 P 3e17a3a82bb24b8fa3e014c9c70c5d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e17a3a82bb24b8fa3e014c9c70c5d91, State: Initialized, Role: FOLLOWER
I20260810 23:25:45.298985 22960 consensus_queue.cc:260] T 442b0cc01faa4855b76655a33f41a874 P 3e17a3a82bb24b8fa3e014c9c70c5d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc3795c0472482f8ada6e12961213ec" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 42033 } } peers { permanent_uuid: "a96dc7f09d9b49069d1d11c24d5b3548" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 34707 } } peers { permanent_uuid: "3e17a3a82bb24b8fa3e014c9c70c5d91" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 35807 } }
I20260810 23:25:45.299268 22959 ts_tablet_manager.cc:1428] T 442b0cc01faa4855b76655a33f41a874 P 3dc3795c0472482f8ada6e12961213ec: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260810 23:25:45.304311 22925 heartbeater.cc:503] Master 127.14.175.126:38057 was elected leader, sending a full tablet report...
I20260810 23:25:45.305564 22958 ts_tablet_manager.cc:1428] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548: Time spent starting tablet: real 0.021s user 0.014s sys 0.008s
I20260810 23:25:45.306628 22717 heartbeater.cc:503] Master 127.14.175.126:38057 was elected leader, sending a full tablet report...
I20260810 23:25:45.307271 22960 ts_tablet_manager.cc:1428] T 442b0cc01faa4855b76655a33f41a874 P 3e17a3a82bb24b8fa3e014c9c70c5d91: Time spent starting tablet: real 0.020s user 0.015s sys 0.006s
W20260810 23:25:45.421523 22928 tablet.cc:2369] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:25:45.458724 22718 tablet.cc:2369] T 442b0cc01faa4855b76655a33f41a874 P 3e17a3a82bb24b8fa3e014c9c70c5d91: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:25:45.492965 22511 tablet.cc:2369] T 442b0cc01faa4855b76655a33f41a874 P 3dc3795c0472482f8ada6e12961213ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:25:45.686825 22966 raft_consensus.cc:482] T 442b0cc01faa4855b76655a33f41a874 P 3dc3795c0472482f8ada6e12961213ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:25:45.687172 22966 raft_consensus.cc:504] T 442b0cc01faa4855b76655a33f41a874 P 3dc3795c0472482f8ada6e12961213ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc3795c0472482f8ada6e12961213ec" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 42033 } } peers { permanent_uuid: "a96dc7f09d9b49069d1d11c24d5b3548" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 34707 } } peers { permanent_uuid: "3e17a3a82bb24b8fa3e014c9c70c5d91" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 35807 } }
I20260810 23:25:45.688984 22966 leader_election.cc:288] T 442b0cc01faa4855b76655a33f41a874 P 3dc3795c0472482f8ada6e12961213ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a96dc7f09d9b49069d1d11c24d5b3548 (127.14.175.67:34707), 3e17a3a82bb24b8fa3e014c9c70c5d91 (127.14.175.66:35807)
I20260810 23:25:45.690068 22967 raft_consensus.cc:482] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:25:45.690508 22967 raft_consensus.cc:504] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc3795c0472482f8ada6e12961213ec" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 42033 } } peers { permanent_uuid: "a96dc7f09d9b49069d1d11c24d5b3548" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 34707 } } peers { permanent_uuid: "3e17a3a82bb24b8fa3e014c9c70c5d91" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 35807 } }
I20260810 23:25:45.702670 22967 leader_election.cc:288] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3dc3795c0472482f8ada6e12961213ec (127.14.175.65:42033), 3e17a3a82bb24b8fa3e014c9c70c5d91 (127.14.175.66:35807)
I20260810 23:25:45.704298 22949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:45.693705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:58603 (local address 127.14.175.65:42033)
0810 23:25:45.693914 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:45.693921 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:45.699045 (+ 5124us) server_negotiation.cc:407] Connection header received
0810 23:25:45.699374 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:45.699381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:45.699459 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:45.699549 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:45.703471 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:45.703477 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:45.703481 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:25:45.703506 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:45.703555 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:25:45.703571 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:45.703686 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:45.703938 (+ 252us) server_negotiation.cc:299] Negotiation successful
0810 23:25:45.704007 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 23:25:45.705240 22971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:45.693129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:42033 (local address 127.14.175.67:58603)
0810 23:25:45.698901 (+ 5772us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:45.698926 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:45.699158 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:45.699792 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:45.699798 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:45.699828 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:45.703196 (+ 3368us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:25:45.703199 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:45.703221 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:25:45.703281 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:45.703788 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:45.703794 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:45.703799 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:25:45.704621 (+ 822us) client_negotiation.cc:240] Negotiation successful
0810 23:25:45.704687 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5619,"thread_start_us":5931,"threads_started":1}
I20260810 23:25:45.705498 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:45.689767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:34061 (local address 127.14.175.67:34707)
0810 23:25:45.689948 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:45.689955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:45.702300 (+ 12345us) server_negotiation.cc:407] Connection header received
0810 23:25:45.702754 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:45.702759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:45.702817 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:45.702889 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:45.704804 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:45.704808 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:45.704811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:25:45.704827 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:45.704858 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:25:45.704869 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:45.704960 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:45.705200 (+ 240us) server_negotiation.cc:299] Negotiation successful
0810 23:25:45.705271 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 23:25:45.706245 22441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442b0cc01faa4855b76655a33f41a874" candidate_uuid: "a96dc7f09d9b49069d1d11c24d5b3548" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dc3795c0472482f8ada6e12961213ec" is_pre_election: true
I20260810 23:25:45.706532 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:45.698458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:38819 (local address 127.14.175.66:35807)
0810 23:25:45.699142 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:45.699148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:45.703867 (+ 4719us) server_negotiation.cc:407] Connection header received
0810 23:25:45.703930 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:45.703934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:45.703982 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:45.704073 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:45.705502 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:45.705507 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:45.705511 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:25:45.705532 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:45.705581 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:25:45.705597 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:45.705747 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:45.705986 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 23:25:45.706059 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":305,"threads_started":1}
I20260810 23:25:45.706791 22956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:45.689616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:54731 (local address 127.14.175.66:35807)
0810 23:25:45.689814 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:45.689820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:45.702554 (+ 12734us) server_negotiation.cc:407] Connection header received
0810 23:25:45.702785 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:45.702790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:45.702847 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:45.702910 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:45.706027 (+ 3117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:45.706033 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:45.706037 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:25:45.706059 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:45.706096 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:25:45.706115 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:45.706230 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:45.706417 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 23:25:45.706472 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 23:25:45.706817 22441 raft_consensus.cc:2454] T 442b0cc01faa4855b76655a33f41a874 P 3dc3795c0472482f8ada6e12961213ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a96dc7f09d9b49069d1d11c24d5b3548 in term 0.
I20260810 23:25:45.707257 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:45.697961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:35807 (local address 127.14.175.67:38819)
0810 23:25:45.703648 (+ 5687us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:45.703668 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:45.703766 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:45.704377 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:45.704382 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:45.704407 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:45.705281 (+ 874us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:25:45.705284 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:45.705301 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:25:45.705341 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:45.705846 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:45.705850 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:45.705854 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:25:45.706799 (+ 945us) client_negotiation.cc:240] Negotiation successful
0810 23:25:45.706861 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5504,"thread_start_us":5347,"threads_started":1}
I20260810 23:25:45.707559 22970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:45.689448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:35807 (local address 127.14.175.65:54731)
0810 23:25:45.702436 (+ 12988us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:45.702455 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:45.702647 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:45.703134 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:45.703139 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:45.703162 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:45.704981 (+ 1819us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:25:45.704984 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:45.705002 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:25:45.705041 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:45.706283 (+ 1242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:45.706288 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:45.706294 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 23:25:45.707032 (+ 738us) client_negotiation.cc:240] Negotiation successful
0810 23:25:45.707098 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12859,"thread_start_us":12857,"threads_started":1}
I20260810 23:25:45.707923 22740 leader_election.cc:302] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dc3795c0472482f8ada6e12961213ec, a96dc7f09d9b49069d1d11c24d5b3548; no voters:
I20260810 23:25:45.708413 22969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:45.689329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:34707 (local address 127.14.175.65:34061)
0810 23:25:45.702156 (+ 12827us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:45.702181 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:45.702602 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:45.703129 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:45.703134 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:45.703161 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:45.704226 (+ 1065us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:25:45.704229 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:45.704248 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:25:45.704303 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:45.705043 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:45.705047 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:45.705051 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:25:45.707908 (+ 2857us) client_negotiation.cc:240] Negotiation successful
0810 23:25:45.707965 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12648,"thread_start_us":12594,"threads_started":1}
I20260810 23:25:45.708424 22967 raft_consensus.cc:2781] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:25:45.708767 22645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442b0cc01faa4855b76655a33f41a874" candidate_uuid: "a96dc7f09d9b49069d1d11c24d5b3548" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e17a3a82bb24b8fa3e014c9c70c5d91" is_pre_election: true
I20260810 23:25:45.708994 22642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442b0cc01faa4855b76655a33f41a874" candidate_uuid: "3dc3795c0472482f8ada6e12961213ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e17a3a82bb24b8fa3e014c9c70c5d91" is_pre_election: true
I20260810 23:25:45.709455 22645 raft_consensus.cc:2454] T 442b0cc01faa4855b76655a33f41a874 P 3e17a3a82bb24b8fa3e014c9c70c5d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a96dc7f09d9b49069d1d11c24d5b3548 in term 0.
I20260810 23:25:45.709444 22838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442b0cc01faa4855b76655a33f41a874" candidate_uuid: "3dc3795c0472482f8ada6e12961213ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a96dc7f09d9b49069d1d11c24d5b3548" is_pre_election: true
I20260810 23:25:45.709026 22967 raft_consensus.cc:482] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:25:45.710668 22967 raft_consensus.cc:3037] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:45.713372 22967 raft_consensus.cc:504] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc3795c0472482f8ada6e12961213ec" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 42033 } } peers { permanent_uuid: "a96dc7f09d9b49069d1d11c24d5b3548" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 34707 } } peers { permanent_uuid: "3e17a3a82bb24b8fa3e014c9c70c5d91" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 35807 } }
I20260810 23:25:45.713910 22838 raft_consensus.cc:2379] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3dc3795c0472482f8ada6e12961213ec in current term 1: Already voted for candidate a96dc7f09d9b49069d1d11c24d5b3548 in this term.
I20260810 23:25:45.714267 22967 leader_election.cc:288] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548 [CANDIDATE]: Term 1 election: Requested vote from peers 3dc3795c0472482f8ada6e12961213ec (127.14.175.65:42033), 3e17a3a82bb24b8fa3e014c9c70c5d91 (127.14.175.66:35807)
I20260810 23:25:45.714522 22441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442b0cc01faa4855b76655a33f41a874" candidate_uuid: "a96dc7f09d9b49069d1d11c24d5b3548" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dc3795c0472482f8ada6e12961213ec"
I20260810 23:25:45.714771 22441 raft_consensus.cc:3037] T 442b0cc01faa4855b76655a33f41a874 P 3dc3795c0472482f8ada6e12961213ec [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:45.714978 22642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442b0cc01faa4855b76655a33f41a874" candidate_uuid: "a96dc7f09d9b49069d1d11c24d5b3548" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e17a3a82bb24b8fa3e014c9c70c5d91"
I20260810 23:25:45.715219 22642 raft_consensus.cc:3037] T 442b0cc01faa4855b76655a33f41a874 P 3e17a3a82bb24b8fa3e014c9c70c5d91 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:45.718159 22642 raft_consensus.cc:2454] T 442b0cc01faa4855b76655a33f41a874 P 3e17a3a82bb24b8fa3e014c9c70c5d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a96dc7f09d9b49069d1d11c24d5b3548 in term 1.
I20260810 23:25:45.718155 22354 leader_election.cc:302] T 442b0cc01faa4855b76655a33f41a874 P 3dc3795c0472482f8ada6e12961213ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3dc3795c0472482f8ada6e12961213ec; no voters: 3e17a3a82bb24b8fa3e014c9c70c5d91, a96dc7f09d9b49069d1d11c24d5b3548
I20260810 23:25:45.718298 22441 raft_consensus.cc:2454] T 442b0cc01faa4855b76655a33f41a874 P 3dc3795c0472482f8ada6e12961213ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a96dc7f09d9b49069d1d11c24d5b3548 in term 1.
I20260810 23:25:45.718637 22739 leader_election.cc:302] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e17a3a82bb24b8fa3e014c9c70c5d91, a96dc7f09d9b49069d1d11c24d5b3548; no voters:
I20260810 23:25:45.719072 22966 raft_consensus.cc:2726] T 442b0cc01faa4855b76655a33f41a874 P 3dc3795c0472482f8ada6e12961213ec [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:25:45.719123 22967 raft_consensus.cc:2781] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:25:45.719584 22967 raft_consensus.cc:686] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548 [term 1 LEADER]: Becoming Leader. State: Replica: a96dc7f09d9b49069d1d11c24d5b3548, State: Running, Role: LEADER
I20260810 23:25:45.720136 22967 consensus_queue.cc:237] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc3795c0472482f8ada6e12961213ec" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 42033 } } peers { permanent_uuid: "a96dc7f09d9b49069d1d11c24d5b3548" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 34707 } } peers { permanent_uuid: "3e17a3a82bb24b8fa3e014c9c70c5d91" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 35807 } }
I20260810 23:25:45.724434 22240 catalog_manager.cc:5495] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548 reported cstate change: term changed from 0 to 1, leader changed from <none> to a96dc7f09d9b49069d1d11c24d5b3548 (127.14.175.67). New cstate: current_term: 1 leader_uuid: "a96dc7f09d9b49069d1d11c24d5b3548" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc3795c0472482f8ada6e12961213ec" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 42033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a96dc7f09d9b49069d1d11c24d5b3548" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 34707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e17a3a82bb24b8fa3e014c9c70c5d91" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 35807 } health_report { overall_health: UNKNOWN } } }
I20260810 23:25:45.833081 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:45.830483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45308 (local address 127.14.175.67:34707)
0810 23:25:45.830677 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:45.830682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:45.831230 (+ 548us) server_negotiation.cc:407] Connection header received
0810 23:25:45.831410 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:45.831413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:45.831523 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:45.831610 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:45.832497 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:45.832502 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:45.832505 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:25:45.832520 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:45.832552 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:25:45.832564 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:45.832700 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:45.832739 (+ 39us) server_negotiation.cc:299] Negotiation successful
0810 23:25:45.832788 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:25:45.841572 22642 raft_consensus.cc:1261] T 442b0cc01faa4855b76655a33f41a874 P 3e17a3a82bb24b8fa3e014c9c70c5d91 [term 1 FOLLOWER]: Refusing update from remote peer a96dc7f09d9b49069d1d11c24d5b3548: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:25:45.841617 22441 raft_consensus.cc:1261] T 442b0cc01faa4855b76655a33f41a874 P 3dc3795c0472482f8ada6e12961213ec [term 1 FOLLOWER]: Refusing update from remote peer a96dc7f09d9b49069d1d11c24d5b3548: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:25:45.842555 22975 consensus_queue.cc:1035] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc3795c0472482f8ada6e12961213ec" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 42033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:25:45.842943 22967 consensus_queue.cc:1035] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e17a3a82bb24b8fa3e014c9c70c5d91" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 35807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:25:45.854838 22986 mvcc.cc:205] Tried to move back new op lower bound from 7317112200556892160 to 7317112200073441280. Current Snapshot: MvccSnapshot[applied={T|T < 7317112200556892160}]
I20260810 23:25:45.869530 22240 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:25:45.869798 22240 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:25:45.873585 22240 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 93cf75a16e734077b891660c9ee175a5: Sending DeleteTablet for 3 replicas of tablet 442b0cc01faa4855b76655a33f41a874
I20260810 23:25:45.875036 22810 tablet_service.cc:1514] Processing DeleteTablet for tablet 442b0cc01faa4855b76655a33f41a874 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:25:45 UTC) from {username='slave'} at 127.0.0.1:45302
I20260810 23:25:45.875002 22412 tablet_service.cc:1514] Processing DeleteTablet for tablet 442b0cc01faa4855b76655a33f41a874 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:25:45 UTC) from {username='slave'} at 127.0.0.1:57358
I20260810 23:25:45.875241 22615 tablet_service.cc:1514] Processing DeleteTablet for tablet 442b0cc01faa4855b76655a33f41a874 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:25:45 UTC) from {username='slave'} at 127.0.0.1:49922
I20260810 23:25:45.877871 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 22324
I20260810 23:25:45.891404 22999 tablet_replica.cc:330] T 442b0cc01faa4855b76655a33f41a874 P 3e17a3a82bb24b8fa3e014c9c70c5d91: stopping tablet replica
I20260810 23:25:45.892014 22999 raft_consensus.cc:2229] T 442b0cc01faa4855b76655a33f41a874 P 3e17a3a82bb24b8fa3e014c9c70c5d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:25:45.892632 22999 raft_consensus.cc:2258] T 442b0cc01faa4855b76655a33f41a874 P 3e17a3a82bb24b8fa3e014c9c70c5d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:25:45.894987 22999 ts_tablet_manager.cc:1905] T 442b0cc01faa4855b76655a33f41a874 P 3e17a3a82bb24b8fa3e014c9c70c5d91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:25:45.903002 22949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:45.879612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57374 (local address 127.14.175.65:42033)
0810 23:25:45.879819 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:45.879827 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:45.899897 (+ 20070us) server_negotiation.cc:407] Connection header received
0810 23:25:45.900523 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:45.900530 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:45.900651 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:45.900726 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:45.902147 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:45.902154 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:45.902159 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:25:45.902182 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:45.902226 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:25:45.902241 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:45.902355 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:45.902611 (+ 256us) server_negotiation.cc:299] Negotiation successful
0810 23:25:45.902683 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 23:25:45.903344 22999 ts_tablet_manager.cc:1918] T 442b0cc01faa4855b76655a33f41a874 P 3e17a3a82bb24b8fa3e014c9c70c5d91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:25:45.903571 22999 log.cc:1196] T 442b0cc01faa4855b76655a33f41a874 P 3e17a3a82bb24b8fa3e014c9c70c5d91: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-1/wal/wals/442b0cc01faa4855b76655a33f41a874
I20260810 23:25:45.904254 22502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:25:45.905341 22997 tablet_replica.cc:330] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548: stopping tablet replica
I20260810 23:25:45.905350 22999 ts_tablet_manager.cc:1939] T 442b0cc01faa4855b76655a33f41a874 P 3e17a3a82bb24b8fa3e014c9c70c5d91: Deleting consensus metadata
I20260810 23:25:45.905937 22997 raft_consensus.cc:2229] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:25:45.906761 22997 raft_consensus.cc:2258] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:25:45.907863 22219 catalog_manager.cc:4841] TS 3e17a3a82bb24b8fa3e014c9c70c5d91 (127.14.175.66:35807): tablet 442b0cc01faa4855b76655a33f41a874 (table test-table [id=a94425ff50a345c8a4c2279f5c8bbdbb]) successfully deleted
I20260810 23:25:45.910878 22997 ts_tablet_manager.cc:1905] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:25:45.917562 22997 ts_tablet_manager.cc:1918] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:25:45.917791 22997 log.cc:1196] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404299514076-15037-0/minicluster-data/ts-2/wal/wals/442b0cc01faa4855b76655a33f41a874
I20260810 23:25:45.918442 22997 ts_tablet_manager.cc:1939] T 442b0cc01faa4855b76655a33f41a874 P a96dc7f09d9b49069d1d11c24d5b3548: Deleting consensus metadata
I20260810 23:25:45.920471 22222 catalog_manager.cc:4841] TS a96dc7f09d9b49069d1d11c24d5b3548 (127.14.175.67:34707): tablet 442b0cc01faa4855b76655a33f41a874 (table test-table [id=a94425ff50a345c8a4c2279f5c8bbdbb]) successfully deleted
I20260810 23:25:46.019428 22998 tablet_replica.cc:330] T 442b0cc01faa4855b76655a33f41a874 P 3dc3795c0472482f8ada6e12961213ec: stopping tablet replica
I20260810 23:25:46.019495 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 22324
W20260810 23:25:46.051684 22220 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.175.65:42033 (error 108)
I20260810 23:25:46.052462 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 22519
I20260810 23:25:46.054095 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:46.053184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:42033 (local address 127.0.0.1:57376)
0810 23:25:46.053708 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:46.053869 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:42033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":555,"threads_started":1}
W20260810 23:25:46.054569 22220 catalog_manager.cc:4584] TS 3dc3795c0472482f8ada6e12961213ec (127.14.175.65:42033): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 442b0cc01faa4855b76655a33f41a874: Network error: Client connection negotiation failed: client connection to 127.14.175.65:42033: connect: Connection refused (error 111)
I20260810 23:25:46.055666 22956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:46.053338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49930 (local address 127.14.175.66:35807)
0810 23:25:46.053529 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:46.053535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:46.053563 (+ 28us) server_negotiation.cc:407] Connection header received
0810 23:25:46.053802 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:46.053808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:46.053924 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:46.054036 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:46.055037 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:46.055042 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:46.055045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:25:46.055062 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:46.055096 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:25:46.055109 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:46.055237 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:46.055332 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 23:25:46.055395 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 23:25:46.056315 22708 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:25:46.108620 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:46.108025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:42033 (local address 127.0.0.1:57382)
0810 23:25:46.108325 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:46.108448 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:42033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260810 23:25:46.109944 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:46.109455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:42033 (local address 127.0.0.1:57396)
0810 23:25:46.109631 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:46.109746 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:42033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 23:25:46.149995 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 22519
I20260810 23:25:46.170997 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:46.170414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:42033 (local address 127.0.0.1:57412)
0810 23:25:46.170704 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:46.170814 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:42033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260810 23:25:46.179334 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:46.171764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:42033 (local address 127.0.0.1:57416)
0810 23:25:46.178905 (+ 7141us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:46.179045 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:42033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7000,"threads_started":1}
I20260810 23:25:46.180995 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 22721
I20260810 23:25:46.184335 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:46.182017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45324 (local address 127.14.175.67:34707)
0810 23:25:46.182224 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:46.182230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:46.182288 (+ 58us) server_negotiation.cc:407] Connection header received
0810 23:25:46.182475 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:46.182481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:46.182554 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:46.182677 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:46.183694 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:46.183701 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:46.183706 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:25:46.183727 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:46.183770 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:25:46.183788 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:46.183937 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:46.183988 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 23:25:46.184050 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 23:25:46.185271 22917 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:25:46.261780 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:46.261289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:42033 (local address 127.0.0.1:57424)
0810 23:25:46.261519 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:46.261628 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:42033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260810 23:25:46.262949 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:46.262474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:42033 (local address 127.0.0.1:57432)
0810 23:25:46.262683 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:46.262767 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:42033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260810 23:25:46.307817 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 22721
I20260810 23:25:46.334828 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 22197
I20260810 23:25:46.336176 22287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:25:46.448120 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 22197
2026-08-10T23:25:46Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3287 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 23:25:46.498390 15037 mini_oidc.cc:152] Starting JWKS server
I20260810 23:25:46.502737 15037 webserver.cc:460] Webserver started at https://127.0.0.1:34915/ using document root <none> and password file <none>
I20260810 23:25:46.503126 15037 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 23:25:46.503790 15037 webserver.cc:460] Webserver started at http://127.0.0.1:35873/ using document root <none> and password file <none>
2026-08-10T23:25:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:25:46Z Disabled control of system clock
I20260810 23:25:46.538264 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:37363
--webserver_interface=127.14.175.126
--webserver_port=0
--builtin_ntp_servers=127.14.175.84:41717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:37363
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34915/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404299514076-15037-0/testchainca.cert with env {}
W20260810 23:25:46.891585 23022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:46.891937 23022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:46.892082 23022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:46.899619 23022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:25:46.899771 23022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:46.899840 23022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:46.899926 23022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:25:46.899988 23022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 23:25:46.900079 23022 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 23:25:46.900136 23022 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34915/jwks.json
I20260810 23:25:46.910739 23022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:41717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:37363
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:37363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34915/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404299514076-15037-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:46.912199 23022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:46.913642 23022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:46.926198 23034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:46.926218 23029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:46.926343 23028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:46.927194 23022 server_base.cc:1034] running on GCE node
I20260810 23:25:46.928087 23022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:46.929692 23022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:46.930902 23022 hybrid_clock.cc:648] HybridClock initialized: now 1786404346930856 us; error 39 us; skew 500 ppm
I20260810 23:25:46.931274 23022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:25:46.933838 23022 webserver.cc:460] Webserver started at http://127.14.175.126:34225/ using document root <none> and password file <none>
I20260810 23:25:46.934536 23022 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:46.934644 23022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:46.934926 23022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:46.938077 23022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "b88d3485c272400e92924c4f52dca923"
format_stamp: "Formatted at 2026-08-10 23:25:46 on dist-test-slave-rmhg"
server_key: "b7dbe3710c3a66b98ddde5db0c055187"
server_key_iv: "26491f66ddf66a093621f868397da437"
server_key_version: "encryptionkey@0"
I20260810 23:25:46.938884 23022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "b88d3485c272400e92924c4f52dca923"
format_stamp: "Formatted at 2026-08-10 23:25:46 on dist-test-slave-rmhg"
server_key: "b7dbe3710c3a66b98ddde5db0c055187"
server_key_iv: "26491f66ddf66a093621f868397da437"
server_key_version: "encryptionkey@0"
I20260810 23:25:46.944396 23022 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:25:46.948342 23045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:46.949880 23022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 23:25:46.950071 23022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "b88d3485c272400e92924c4f52dca923"
format_stamp: "Formatted at 2026-08-10 23:25:46 on dist-test-slave-rmhg"
server_key: "b7dbe3710c3a66b98ddde5db0c055187"
server_key_iv: "26491f66ddf66a093621f868397da437"
server_key_version: "encryptionkey@0"
I20260810 23:25:46.950297 23022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:46.971626 23022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:46.972545 23022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:46.972870 23022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:46.991379 23022 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:37363
I20260810 23:25:46.991420 23127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:37363 every 8 connection(s)
I20260810 23:25:46.992811 23022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:25:46.996860 23128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:47.002712 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 23022
I20260810 23:25:47.002925 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:25:47.003815 15037 external_mini_cluster.cc:1428] Setting key 9df1c95b26104c93a7f7cff1262f7bad
I20260810 23:25:47.009203 23128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b88d3485c272400e92924c4f52dca923: Bootstrap starting.
I20260810 23:25:47.013137 23128 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b88d3485c272400e92924c4f52dca923: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:47.014580 23128 log.cc:824] T 00000000000000000000000000000000 P b88d3485c272400e92924c4f52dca923: Log is configured to *not* fsync() on all Append() calls
I20260810 23:25:47.018260 23128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b88d3485c272400e92924c4f52dca923: No bootstrap required, opened a new log
I20260810 23:25:47.019423 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:47.006026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58676 (local address 127.14.175.126:37363)
0810 23:25:47.007278 (+ 1252us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:47.007289 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:47.007337 (+ 48us) server_negotiation.cc:407] Connection header received
0810 23:25:47.009138 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:47.009205 (+ 67us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:47.009773 (+ 568us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:47.010149 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:47.010883 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:47.012286 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:47.014237 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:47.015019 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:47.015264 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:47.016413 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:47.016424 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:47.016445 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:25:47.017223 (+ 778us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:47.017337 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:25:47.017366 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:47.017492 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:47.017697 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 23:25:47.018060 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":753,"thread_start_us":470,"threads_started":1}
I20260810 23:25:47.023991 15037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:25:47.027231 23128 raft_consensus.cc:348] T 00000000000000000000000000000000 P b88d3485c272400e92924c4f52dca923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b88d3485c272400e92924c4f52dca923" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37363 } }
I20260810 23:25:47.027652 23128 raft_consensus.cc:374] T 00000000000000000000000000000000 P b88d3485c272400e92924c4f52dca923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:47.027745 23128 raft_consensus.cc:729] T 00000000000000000000000000000000 P b88d3485c272400e92924c4f52dca923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b88d3485c272400e92924c4f52dca923, State: Initialized, Role: FOLLOWER
I20260810 23:25:47.028311 23128 consensus_queue.cc:260] T 00000000000000000000000000000000 P b88d3485c272400e92924c4f52dca923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b88d3485c272400e92924c4f52dca923" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37363 } }
I20260810 23:25:47.028532 23128 raft_consensus.cc:388] T 00000000000000000000000000000000 P b88d3485c272400e92924c4f52dca923 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:25:47.028662 23128 raft_consensus.cc:482] T 00000000000000000000000000000000 P b88d3485c272400e92924c4f52dca923 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:25:47.028816 23128 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b88d3485c272400e92924c4f52dca923 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:47.031338 23128 raft_consensus.cc:504] T 00000000000000000000000000000000 P b88d3485c272400e92924c4f52dca923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b88d3485c272400e92924c4f52dca923" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37363 } }
I20260810 23:25:47.031854 23128 leader_election.cc:302] T 00000000000000000000000000000000 P b88d3485c272400e92924c4f52dca923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b88d3485c272400e92924c4f52dca923; no voters:
I20260810 23:25:47.032423 23128 leader_election.cc:288] T 00000000000000000000000000000000 P b88d3485c272400e92924c4f52dca923 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:25:47.032558 23135 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b88d3485c272400e92924c4f52dca923 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:25:47.033308 23135 raft_consensus.cc:686] T 00000000000000000000000000000000 P b88d3485c272400e92924c4f52dca923 [term 1 LEADER]: Becoming Leader. State: Replica: b88d3485c272400e92924c4f52dca923, State: Running, Role: LEADER
I20260810 23:25:47.033833 23135 consensus_queue.cc:237] T 00000000000000000000000000000000 P b88d3485c272400e92924c4f52dca923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b88d3485c272400e92924c4f52dca923" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37363 } }
I20260810 23:25:47.034493 23128 sys_catalog.cc:564] T 00000000000000000000000000000000 P b88d3485c272400e92924c4f52dca923 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:25:47.036675 23137 sys_catalog.cc:455] T 00000000000000000000000000000000 P b88d3485c272400e92924c4f52dca923 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b88d3485c272400e92924c4f52dca923. Latest consensus state: current_term: 1 leader_uuid: "b88d3485c272400e92924c4f52dca923" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b88d3485c272400e92924c4f52dca923" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37363 } } }
I20260810 23:25:47.036545 23136 sys_catalog.cc:455] T 00000000000000000000000000000000 P b88d3485c272400e92924c4f52dca923 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b88d3485c272400e92924c4f52dca923" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b88d3485c272400e92924c4f52dca923" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37363 } } }
I20260810 23:25:47.037217 23136 sys_catalog.cc:458] T 00000000000000000000000000000000 P b88d3485c272400e92924c4f52dca923 [sys.catalog]: This master's current role is: LEADER
I20260810 23:25:47.038443 23137 sys_catalog.cc:458] T 00000000000000000000000000000000 P b88d3485c272400e92924c4f52dca923 [sys.catalog]: This master's current role is: LEADER
I20260810 23:25:47.039970 23144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:25:47.045850 23144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:25:47.055755 23144 catalog_manager.cc:1347] Generated new cluster ID: d35c71dec57d4098b2db634445975988
I20260810 23:25:47.055874 23144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:25:47.067873 23144 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:25:47.069480 23144 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:25:47.085425 23144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b88d3485c272400e92924c4f52dca923: Generated new TSK 0
I20260810 23:25:47.086380 23144 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 23:25:47.131174 23171 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 23:25:47.131412 23171 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 23:25:47.134016 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:47.108607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58682 (local address 127.14.175.126:37363)
0810 23:25:47.108821 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:47.108828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:47.109389 (+ 561us) server_negotiation.cc:407] Connection header received
0810 23:25:47.109792 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:47.109798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:47.109902 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:47.110071 (+ 169us) server_negotiation.cc:226] Negotiated authn=JWT
0810 23:25:47.110666 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:47.111833 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:47.113352 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:47.114092 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:47.114234 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:47.114787 (+ 553us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 23:25:47.133214 (+ 18427us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34915/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 23:25:47.133691 (+ 477us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58682: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34915/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":73}
W20260810 23:25:47.134158 23133 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58682: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34915/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260810 23:25:47.135787 23170 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:25:47.108557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:37363 (local address 127.0.0.1:58682)
0810 23:25:47.109287 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:47.109304 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:47.109476 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:47.110140 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:47.110145 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:47.110156 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0810 23:25:47.110495 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:47.110503 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:47.111996 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:47.112000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:47.113196 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:47.113215 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:47.114268 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:47.114272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:47.114409 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:47.114483 (+ 74us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 23:25:47.133562 (+ 19079us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34915/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 23:25:47.133901 (+ 339us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.175.126:37363: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34915/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":342,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.14.175.126:37363: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34915/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 23:25:47.137745 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 23022
I20260810 23:25:47.139007 23120 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:25:47.237924 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 23022
2026-08-10T23:25:47Z chronyd exiting
I20260810 23:25:47.296790 15037 test_util.cc:175] -----------------------------------------------
I20260810 23:25:47.296939 15037 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404299514076-15037-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (803 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 23:25:47.300985 15037 mini_oidc.cc:152] Starting JWKS server
I20260810 23:25:47.302997 15037 webserver.cc:460] Webserver started at https://127.0.0.1:45255/ using document root <none> and password file <none>
I20260810 23:25:47.303268 15037 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 23:25:47.304005 15037 webserver.cc:460] Webserver started at http://127.0.0.1:39941/ using document root <none> and password file <none>
2026-08-10T23:25:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:25:47Z Disabled control of system clock
I20260810 23:25:47.339013 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:36611
--webserver_interface=127.14.175.126
--webserver_port=0
--builtin_ntp_servers=127.14.175.84:33533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:36611
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45255/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404299514076-15037-0/testchainca.cert with env {}
W20260810 23:25:47.686314 23184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:47.686651 23184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:47.686787 23184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:47.694429 23184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:25:47.694590 23184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:47.694659 23184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:47.694746 23184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:25:47.694806 23184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 23:25:47.694897 23184 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 23:25:47.694945 23184 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45255/jwks.json
I20260810 23:25:47.705713 23184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:33533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:36611
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:36611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45255/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404299514076-15037-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:47.707183 23184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:47.708580 23184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:47.720749 23191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:47.720827 23192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:47.720777 23194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:47.722277 23184 server_base.cc:1034] running on GCE node
I20260810 23:25:47.723114 23184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:47.724675 23184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:47.725929 23184 hybrid_clock.cc:648] HybridClock initialized: now 1786404347725867 us; error 48 us; skew 500 ppm
I20260810 23:25:47.726296 23184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:25:47.733760 23184 webserver.cc:460] Webserver started at http://127.14.175.126:36217/ using document root <none> and password file <none>
I20260810 23:25:47.734571 23184 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:47.734688 23184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:47.735004 23184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:47.738340 23184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "8bd1127480114e7cbaaa29aaad1453e0"
format_stamp: "Formatted at 2026-08-10 23:25:47 on dist-test-slave-rmhg"
server_key: "c9efc220080fb077e7fe833a392ff86b"
server_key_iv: "5569585b67ad63d8f5d93188335787ac"
server_key_version: "encryptionkey@0"
I20260810 23:25:47.739250 23184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "8bd1127480114e7cbaaa29aaad1453e0"
format_stamp: "Formatted at 2026-08-10 23:25:47 on dist-test-slave-rmhg"
server_key: "c9efc220080fb077e7fe833a392ff86b"
server_key_iv: "5569585b67ad63d8f5d93188335787ac"
server_key_version: "encryptionkey@0"
I20260810 23:25:47.745301 23184 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260810 23:25:47.749202 23201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:47.750819 23184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:25:47.751008 23184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "8bd1127480114e7cbaaa29aaad1453e0"
format_stamp: "Formatted at 2026-08-10 23:25:47 on dist-test-slave-rmhg"
server_key: "c9efc220080fb077e7fe833a392ff86b"
server_key_iv: "5569585b67ad63d8f5d93188335787ac"
server_key_version: "encryptionkey@0"
I20260810 23:25:47.751211 23184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:47.777319 23184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:47.778252 23184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:47.778564 23184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:47.797719 23184 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:36611
I20260810 23:25:47.797745 23280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:36611 every 8 connection(s)
I20260810 23:25:47.799196 23184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:25:47.803215 23281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:47.803776 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 23184
I20260810 23:25:47.803965 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:25:47.804629 15037 external_mini_cluster.cc:1428] Setting key e3c5e80a22259a5dcdd4a9101305d241
I20260810 23:25:47.817974 23281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bd1127480114e7cbaaa29aaad1453e0: Bootstrap starting.
I20260810 23:25:47.819455 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:47.806420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45958 (local address 127.14.175.126:36611)
0810 23:25:47.807556 (+ 1136us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:47.807569 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:47.807619 (+ 50us) server_negotiation.cc:407] Connection header received
0810 23:25:47.810012 (+ 2393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:47.810042 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:47.810612 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:47.810987 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:47.811572 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:47.812878 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:47.814761 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:47.815539 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:47.815685 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:47.816977 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:47.816985 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:47.817002 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:25:47.817624 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:47.817718 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:25:47.817741 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:47.817836 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:47.818082 (+ 246us) server_negotiation.cc:299] Negotiation successful
0810 23:25:47.818407 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":432,"threads_started":1}
I20260810 23:25:47.821801 23281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bd1127480114e7cbaaa29aaad1453e0: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:47.823035 23281 log.cc:824] T 00000000000000000000000000000000 P 8bd1127480114e7cbaaa29aaad1453e0: Log is configured to *not* fsync() on all Append() calls
I20260810 23:25:47.823769 15037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:25:47.826191 23281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bd1127480114e7cbaaa29aaad1453e0: No bootstrap required, opened a new log
I20260810 23:25:47.834311 23281 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bd1127480114e7cbaaa29aaad1453e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd1127480114e7cbaaa29aaad1453e0" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 36611 } }
I20260810 23:25:47.834736 23281 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bd1127480114e7cbaaa29aaad1453e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:47.834833 23281 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bd1127480114e7cbaaa29aaad1453e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bd1127480114e7cbaaa29aaad1453e0, State: Initialized, Role: FOLLOWER
I20260810 23:25:47.835518 23281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bd1127480114e7cbaaa29aaad1453e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd1127480114e7cbaaa29aaad1453e0" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 36611 } }
I20260810 23:25:47.835868 23281 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bd1127480114e7cbaaa29aaad1453e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:25:47.836069 23281 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bd1127480114e7cbaaa29aaad1453e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:25:47.836239 23281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bd1127480114e7cbaaa29aaad1453e0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:47.838581 23281 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bd1127480114e7cbaaa29aaad1453e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd1127480114e7cbaaa29aaad1453e0" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 36611 } }
I20260810 23:25:47.839133 23281 leader_election.cc:302] T 00000000000000000000000000000000 P 8bd1127480114e7cbaaa29aaad1453e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bd1127480114e7cbaaa29aaad1453e0; no voters:
I20260810 23:25:47.839753 23281 leader_election.cc:288] T 00000000000000000000000000000000 P 8bd1127480114e7cbaaa29aaad1453e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:25:47.839910 23287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bd1127480114e7cbaaa29aaad1453e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:25:47.841056 23287 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bd1127480114e7cbaaa29aaad1453e0 [term 1 LEADER]: Becoming Leader. State: Replica: 8bd1127480114e7cbaaa29aaad1453e0, State: Running, Role: LEADER
I20260810 23:25:47.841775 23287 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bd1127480114e7cbaaa29aaad1453e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd1127480114e7cbaaa29aaad1453e0" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 36611 } }
I20260810 23:25:47.842026 23281 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bd1127480114e7cbaaa29aaad1453e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:25:47.844735 23288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd1127480114e7cbaaa29aaad1453e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bd1127480114e7cbaaa29aaad1453e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd1127480114e7cbaaa29aaad1453e0" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 36611 } } }
I20260810 23:25:47.845103 23288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd1127480114e7cbaaa29aaad1453e0 [sys.catalog]: This master's current role is: LEADER
I20260810 23:25:47.845628 23290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd1127480114e7cbaaa29aaad1453e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bd1127480114e7cbaaa29aaad1453e0. Latest consensus state: current_term: 1 leader_uuid: "8bd1127480114e7cbaaa29aaad1453e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd1127480114e7cbaaa29aaad1453e0" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 36611 } } }
I20260810 23:25:47.845968 23290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd1127480114e7cbaaa29aaad1453e0 [sys.catalog]: This master's current role is: LEADER
I20260810 23:25:47.847409 23301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:25:47.852615 23301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:25:47.862661 23301 catalog_manager.cc:1347] Generated new cluster ID: 6bb9df38225645d7983f587cc01acfd1
I20260810 23:25:47.862807 23301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:25:47.903270 23301 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:25:47.905094 23301 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:25:47.911201 23301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bd1127480114e7cbaaa29aaad1453e0: Generated new TSK 0
I20260810 23:25:47.912370 23301 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 23:25:47.951037 23323 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 23:25:47.951182 23323 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 23:25:47.952310 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:47.931492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45962 (local address 127.14.175.126:36611)
0810 23:25:47.931718 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:47.931728 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:47.932093 (+ 365us) server_negotiation.cc:407] Connection header received
0810 23:25:47.932437 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:47.932442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:47.932520 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:47.932717 (+ 197us) server_negotiation.cc:226] Negotiated authn=JWT
0810 23:25:47.933309 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:47.934120 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:47.936200 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:47.936677 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:47.936784 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:47.937476 (+ 692us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 23:25:47.951701 (+ 14225us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45255/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 23:25:47.952072 (+ 371us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45962: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45255/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":85}
W20260810 23:25:47.952450 23285 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45962: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45255/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260810 23:25:47.952766 23322 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:25:47.931215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:36611 (local address 127.0.0.1:45962)
0810 23:25:47.931973 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:47.931995 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:47.932163 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:47.932746 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:47.932751 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:47.932765 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0810 23:25:47.933104 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:47.933117 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:47.934284 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:47.934290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:47.936010 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:47.936030 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:47.936859 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:47.936866 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:47.937074 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:47.937124 (+ 50us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 23:25:47.951987 (+ 14863us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45255/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 23:25:47.952318 (+ 331us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.175.126:36611: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45255/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":379,"threads_started":1}
I20260810 23:25:47.954101 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 23184
I20260810 23:25:47.955312 23275 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:25:48.051757 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 23184
2026-08-10T23:25:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (812 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 23:25:48.113695 15037 mini_oidc.cc:152] Starting JWKS server
I20260810 23:25:48.115710 15037 webserver.cc:460] Webserver started at https://127.0.0.1:39297/ using document root <none> and password file <none>
I20260810 23:25:48.115954 15037 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 23:25:48.116595 15037 webserver.cc:460] Webserver started at http://127.0.0.1:43365/ using document root <none> and password file <none>
2026-08-10T23:25:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:25:48Z Disabled control of system clock
I20260810 23:25:48.152763 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:41043
--webserver_interface=127.14.175.126
--webserver_port=0
--builtin_ntp_servers=127.14.175.84:37303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:41043
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39297/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 23:25:48.503393 23335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:48.503736 23335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:48.503865 23335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:48.511365 23335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:25:48.511505 23335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:48.511579 23335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:48.511664 23335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:25:48.511726 23335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 23:25:48.511814 23335 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 23:25:48.511868 23335 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39297/jwks.json
I20260810 23:25:48.521553 23335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:37303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:41043
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:41043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39297/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:48.522995 23335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:48.524923 23335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:48.537408 23342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:48.537417 23348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:48.537386 23343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:48.538199 23335 server_base.cc:1034] running on GCE node
I20260810 23:25:48.539091 23335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:48.540650 23335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:48.541867 23335 hybrid_clock.cc:648] HybridClock initialized: now 1786404348541826 us; error 34 us; skew 500 ppm
I20260810 23:25:48.542243 23335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:25:48.544726 23335 webserver.cc:460] Webserver started at http://127.14.175.126:46463/ using document root <none> and password file <none>
I20260810 23:25:48.545519 23335 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:48.545639 23335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:48.545976 23335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:48.549065 23335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "c398543065e24e0dbdab7aadebefc8f6"
format_stamp: "Formatted at 2026-08-10 23:25:48 on dist-test-slave-rmhg"
server_key: "074369a78e584b4fc85ea6b6f25a2e26"
server_key_iv: "2a6ae528da073cb3c3b27f1f4d734b6e"
server_key_version: "encryptionkey@0"
I20260810 23:25:48.549970 23335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "c398543065e24e0dbdab7aadebefc8f6"
format_stamp: "Formatted at 2026-08-10 23:25:48 on dist-test-slave-rmhg"
server_key: "074369a78e584b4fc85ea6b6f25a2e26"
server_key_iv: "2a6ae528da073cb3c3b27f1f4d734b6e"
server_key_version: "encryptionkey@0"
I20260810 23:25:48.555284 23335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:25:48.559195 23355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:48.560680 23335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:25:48.560855 23335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "c398543065e24e0dbdab7aadebefc8f6"
format_stamp: "Formatted at 2026-08-10 23:25:48 on dist-test-slave-rmhg"
server_key: "074369a78e584b4fc85ea6b6f25a2e26"
server_key_iv: "2a6ae528da073cb3c3b27f1f4d734b6e"
server_key_version: "encryptionkey@0"
I20260810 23:25:48.561084 23335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:48.578075 23335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:48.579079 23335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:48.579421 23335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:48.599017 23335 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:41043
I20260810 23:25:48.599040 23432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:41043 every 8 connection(s)
I20260810 23:25:48.600533 23335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:25:48.604694 23433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:48.609766 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 23335
I20260810 23:25:48.609930 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:25:48.610575 15037 external_mini_cluster.cc:1428] Setting key 2d69438da4726165e2748c9cd870040c
I20260810 23:25:48.616802 23433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c398543065e24e0dbdab7aadebefc8f6: Bootstrap starting.
I20260810 23:25:48.620157 23433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c398543065e24e0dbdab7aadebefc8f6: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:48.621376 23433 log.cc:824] T 00000000000000000000000000000000 P c398543065e24e0dbdab7aadebefc8f6: Log is configured to *not* fsync() on all Append() calls
I20260810 23:25:48.624577 23433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c398543065e24e0dbdab7aadebefc8f6: No bootstrap required, opened a new log
I20260810 23:25:48.625403 23436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:48.612597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44292 (local address 127.14.175.126:41043)
0810 23:25:48.613872 (+ 1275us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:48.613885 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:48.613943 (+ 58us) server_negotiation.cc:407] Connection header received
0810 23:25:48.615804 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:48.615837 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:48.616409 (+ 572us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:48.616848 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:48.617674 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:48.619094 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:48.620520 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:48.621340 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:48.621539 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:48.622740 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:48.622749 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:48.622767 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:25:48.623456 (+ 689us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:48.623552 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:25:48.623579 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:48.623688 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:48.623921 (+ 233us) server_negotiation.cc:299] Negotiation successful
0810 23:25:48.624252 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":472,"threads_started":1}
I20260810 23:25:48.629413 15037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:25:48.632967 23433 raft_consensus.cc:348] T 00000000000000000000000000000000 P c398543065e24e0dbdab7aadebefc8f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c398543065e24e0dbdab7aadebefc8f6" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 41043 } }
I20260810 23:25:48.633407 23433 raft_consensus.cc:374] T 00000000000000000000000000000000 P c398543065e24e0dbdab7aadebefc8f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:48.633512 23433 raft_consensus.cc:729] T 00000000000000000000000000000000 P c398543065e24e0dbdab7aadebefc8f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c398543065e24e0dbdab7aadebefc8f6, State: Initialized, Role: FOLLOWER
I20260810 23:25:48.634119 23433 consensus_queue.cc:260] T 00000000000000000000000000000000 P c398543065e24e0dbdab7aadebefc8f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c398543065e24e0dbdab7aadebefc8f6" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 41043 } }
I20260810 23:25:48.634347 23433 raft_consensus.cc:388] T 00000000000000000000000000000000 P c398543065e24e0dbdab7aadebefc8f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:25:48.634476 23433 raft_consensus.cc:482] T 00000000000000000000000000000000 P c398543065e24e0dbdab7aadebefc8f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:25:48.634639 23433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c398543065e24e0dbdab7aadebefc8f6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:48.637091 23433 raft_consensus.cc:504] T 00000000000000000000000000000000 P c398543065e24e0dbdab7aadebefc8f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c398543065e24e0dbdab7aadebefc8f6" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 41043 } }
I20260810 23:25:48.637641 23433 leader_election.cc:302] T 00000000000000000000000000000000 P c398543065e24e0dbdab7aadebefc8f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c398543065e24e0dbdab7aadebefc8f6; no voters:
I20260810 23:25:48.638252 23433 leader_election.cc:288] T 00000000000000000000000000000000 P c398543065e24e0dbdab7aadebefc8f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:25:48.638396 23438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c398543065e24e0dbdab7aadebefc8f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:25:48.639097 23438 raft_consensus.cc:686] T 00000000000000000000000000000000 P c398543065e24e0dbdab7aadebefc8f6 [term 1 LEADER]: Becoming Leader. State: Replica: c398543065e24e0dbdab7aadebefc8f6, State: Running, Role: LEADER
I20260810 23:25:48.639621 23438 consensus_queue.cc:237] T 00000000000000000000000000000000 P c398543065e24e0dbdab7aadebefc8f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c398543065e24e0dbdab7aadebefc8f6" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 41043 } }
I20260810 23:25:48.640426 23433 sys_catalog.cc:564] T 00000000000000000000000000000000 P c398543065e24e0dbdab7aadebefc8f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:25:48.642810 23440 sys_catalog.cc:455] T 00000000000000000000000000000000 P c398543065e24e0dbdab7aadebefc8f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c398543065e24e0dbdab7aadebefc8f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c398543065e24e0dbdab7aadebefc8f6" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 41043 } } }
I20260810 23:25:48.643221 23440 sys_catalog.cc:458] T 00000000000000000000000000000000 P c398543065e24e0dbdab7aadebefc8f6 [sys.catalog]: This master's current role is: LEADER
I20260810 23:25:48.642787 23441 sys_catalog.cc:455] T 00000000000000000000000000000000 P c398543065e24e0dbdab7aadebefc8f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c398543065e24e0dbdab7aadebefc8f6. Latest consensus state: current_term: 1 leader_uuid: "c398543065e24e0dbdab7aadebefc8f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c398543065e24e0dbdab7aadebefc8f6" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 41043 } } }
I20260810 23:25:48.643682 23441 sys_catalog.cc:458] T 00000000000000000000000000000000 P c398543065e24e0dbdab7aadebefc8f6 [sys.catalog]: This master's current role is: LEADER
I20260810 23:25:48.650607 23452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:25:48.656090 23452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:25:48.665684 23452 catalog_manager.cc:1347] Generated new cluster ID: 90fe53e12feb48ccb1d59536d8c6ccb4
I20260810 23:25:48.665814 23452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:25:48.685017 23452 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:25:48.686930 23452 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:25:48.699326 23452 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c398543065e24e0dbdab7aadebefc8f6: Generated new TSK 0
I20260810 23:25:48.700556 23452 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:25:48.726752 23436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:48.717294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44296 (local address 127.14.175.126:41043)
0810 23:25:48.717522 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:48.717529 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:48.717975 (+ 446us) server_negotiation.cc:407] Connection header received
0810 23:25:48.718231 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:48.718238 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:48.718350 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:48.718558 (+ 208us) server_negotiation.cc:226] Negotiated authn=JWT
0810 23:25:48.719104 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:48.720339 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:48.721906 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:48.722609 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:48.722748 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:48.723370 (+ 622us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 23:25:48.726002 (+ 2632us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39297/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 23:25:48.726371 (+ 369us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44296: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39297/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":71}
W20260810 23:25:48.726898 23436 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44296: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39297/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260810 23:25:48.728425 23473 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:25:48.717104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:41043 (local address 127.0.0.1:44296)
0810 23:25:48.717860 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:48.717879 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:48.718024 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:48.718603 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:48.718607 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:48.718617 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0810 23:25:48.718937 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:48.718945 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:48.720524 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:48.720528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:48.721747 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:48.721763 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:48.722853 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:48.722859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:48.723030 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:48.723068 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 23:25:48.726257 (+ 3189us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39297/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 23:25:48.727962 (+ 1705us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.175.126:41043: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39297/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":1374,"thread_start_us":424,"threads_started":1}
I20260810 23:25:48.729615 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 23335
I20260810 23:25:48.731570 23427 generic_service.cc:196] Checking for leaks (request via RPC)
E20260810 23:25:48.738903 23474 webserver.cc:533] Webserver: sslize failed:
E20260810 23:25:48.739025 23474 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 23:25:48.826009 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 23335
2026-08-10T23:25:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (778 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T23:25:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:25:48Z Disabled control of system clock
I20260810 23:25:48.925426 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:43959
--webserver_interface=127.14.175.126
--webserver_port=0
--builtin_ntp_servers=127.14.175.84:39355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:43959
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404299514076-15037-0/testchainca.cert with env {}
W20260810 23:25:49.272352 23483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:49.272698 23483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:49.272795 23483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:49.280180 23483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:25:49.280304 23483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:49.280344 23483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:49.280422 23483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:25:49.280491 23483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 23:25:49.280581 23483 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 23:25:49.280637 23483 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 23:25:49.291024 23483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:39355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:43959
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:43959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404299514076-15037-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:49.292428 23483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:49.293926 23483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:49.309415 23491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:49.309453 23495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:49.309475 23490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:49.310374 23483 server_base.cc:1034] running on GCE node
I20260810 23:25:49.311206 23483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:49.312672 23483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:49.313879 23483 hybrid_clock.cc:648] HybridClock initialized: now 1786404349313823 us; error 44 us; skew 500 ppm
I20260810 23:25:49.314248 23483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:25:49.316653 23483 webserver.cc:460] Webserver started at http://127.14.175.126:42965/ using document root <none> and password file <none>
I20260810 23:25:49.317430 23483 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:49.317545 23483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:49.317858 23483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:49.321254 23483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "685e2543badf4a5c9395f8339cba8430"
format_stamp: "Formatted at 2026-08-10 23:25:49 on dist-test-slave-rmhg"
server_key: "ca66f174a881dcaa693270f5443aabe7"
server_key_iv: "663ba64c37a6c050766acd8368af35b3"
server_key_version: "encryptionkey@0"
I20260810 23:25:49.322142 23483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "685e2543badf4a5c9395f8339cba8430"
format_stamp: "Formatted at 2026-08-10 23:25:49 on dist-test-slave-rmhg"
server_key: "ca66f174a881dcaa693270f5443aabe7"
server_key_iv: "663ba64c37a6c050766acd8368af35b3"
server_key_version: "encryptionkey@0"
I20260810 23:25:49.327714 23483 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260810 23:25:49.331337 23503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:49.332734 23483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:25:49.332903 23483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "685e2543badf4a5c9395f8339cba8430"
format_stamp: "Formatted at 2026-08-10 23:25:49 on dist-test-slave-rmhg"
server_key: "ca66f174a881dcaa693270f5443aabe7"
server_key_iv: "663ba64c37a6c050766acd8368af35b3"
server_key_version: "encryptionkey@0"
I20260810 23:25:49.333125 23483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:49.348249 23483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:49.349197 23483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:49.349478 23483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:49.367887 23483 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:43959
I20260810 23:25:49.367909 23586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:43959 every 8 connection(s)
I20260810 23:25:49.369386 23483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:25:49.373447 23587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:49.379467 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 23483
I20260810 23:25:49.379671 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:25:49.380326 15037 external_mini_cluster.cc:1428] Setting key e04cdb5e82abf68043185adf6e1081cd
I20260810 23:25:49.384989 23587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 685e2543badf4a5c9395f8339cba8430: Bootstrap starting.
I20260810 23:25:49.388665 23587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 685e2543badf4a5c9395f8339cba8430: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:49.390196 23587 log.cc:824] T 00000000000000000000000000000000 P 685e2543badf4a5c9395f8339cba8430: Log is configured to *not* fsync() on all Append() calls
I20260810 23:25:49.394327 23587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 685e2543badf4a5c9395f8339cba8430: No bootstrap required, opened a new log
I20260810 23:25:49.396934 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:49.382425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48660 (local address 127.14.175.126:43959)
0810 23:25:49.383507 (+ 1082us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:49.383518 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:49.383602 (+ 84us) server_negotiation.cc:407] Connection header received
0810 23:25:49.385395 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:49.385420 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:49.385924 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:49.386304 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:49.386900 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:49.388561 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:49.390554 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:49.391741 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:49.391941 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:49.393627 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:49.393638 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:49.393659 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:25:49.394415 (+ 756us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:49.394523 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:25:49.394551 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:49.394646 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:49.395322 (+ 676us) server_negotiation.cc:299] Negotiation successful
0810 23:25:49.395692 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":356,"threads_started":1}
I20260810 23:25:49.403769 15037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:25:49.405536 23587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 685e2543badf4a5c9395f8339cba8430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "685e2543badf4a5c9395f8339cba8430" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43959 } }
I20260810 23:25:49.406037 23587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 685e2543badf4a5c9395f8339cba8430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:49.406162 23587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 685e2543badf4a5c9395f8339cba8430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685e2543badf4a5c9395f8339cba8430, State: Initialized, Role: FOLLOWER
I20260810 23:25:49.406771 23587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 685e2543badf4a5c9395f8339cba8430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "685e2543badf4a5c9395f8339cba8430" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43959 } }
I20260810 23:25:49.407006 23587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 685e2543badf4a5c9395f8339cba8430 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:25:49.407145 23587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 685e2543badf4a5c9395f8339cba8430 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:25:49.407305 23587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 685e2543badf4a5c9395f8339cba8430 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:49.409682 23587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 685e2543badf4a5c9395f8339cba8430 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "685e2543badf4a5c9395f8339cba8430" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43959 } }
I20260810 23:25:49.410260 23587 leader_election.cc:302] T 00000000000000000000000000000000 P 685e2543badf4a5c9395f8339cba8430 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 685e2543badf4a5c9395f8339cba8430; no voters:
I20260810 23:25:49.410998 23587 leader_election.cc:288] T 00000000000000000000000000000000 P 685e2543badf4a5c9395f8339cba8430 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:25:49.411130 23594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 685e2543badf4a5c9395f8339cba8430 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:25:49.411907 23594 raft_consensus.cc:686] T 00000000000000000000000000000000 P 685e2543badf4a5c9395f8339cba8430 [term 1 LEADER]: Becoming Leader. State: Replica: 685e2543badf4a5c9395f8339cba8430, State: Running, Role: LEADER
I20260810 23:25:49.412516 23594 consensus_queue.cc:237] T 00000000000000000000000000000000 P 685e2543badf4a5c9395f8339cba8430 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "685e2543badf4a5c9395f8339cba8430" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43959 } }
I20260810 23:25:49.413635 23587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 685e2543badf4a5c9395f8339cba8430 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:25:49.415722 23596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 685e2543badf4a5c9395f8339cba8430 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 685e2543badf4a5c9395f8339cba8430. Latest consensus state: current_term: 1 leader_uuid: "685e2543badf4a5c9395f8339cba8430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "685e2543badf4a5c9395f8339cba8430" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43959 } } }
I20260810 23:25:49.415791 23595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 685e2543badf4a5c9395f8339cba8430 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "685e2543badf4a5c9395f8339cba8430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "685e2543badf4a5c9395f8339cba8430" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43959 } } }
I20260810 23:25:49.416167 23596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 685e2543badf4a5c9395f8339cba8430 [sys.catalog]: This master's current role is: LEADER
I20260810 23:25:49.416411 23595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 685e2543badf4a5c9395f8339cba8430 [sys.catalog]: This master's current role is: LEADER
I20260810 23:25:49.422206 23607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:25:49.427573 23607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:25:49.436466 23607 catalog_manager.cc:1347] Generated new cluster ID: 2a17f4c7816a4828b34cbea210f379c1
I20260810 23:25:49.436618 23607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:25:49.468475 23607 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:25:49.469910 23607 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:25:49.484261 23607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 685e2543badf4a5c9395f8339cba8430: Generated new TSK 0
I20260810 23:25:49.485345 23607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:25:49.978732 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:49.511634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48666 (local address 127.14.175.126:43959)
0810 23:25:49.511904 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:49.511911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:49.512318 (+ 407us) server_negotiation.cc:407] Connection header received
0810 23:25:49.512635 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:49.512640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:49.512728 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:49.512894 (+ 166us) server_negotiation.cc:226] Negotiated authn=JWT
0810 23:25:49.513522 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:49.514450 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:49.516679 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:49.517267 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:49.517375 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:49.518017 (+ 642us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 23:25:49.978117 (+460100us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 23:25:49.978476 (+ 359us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48666: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":91}
W20260810 23:25:49.978852 23590 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48666: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260810 23:25:49.979100 23631 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:25:49.511424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43959 (local address 127.0.0.1:48666)
0810 23:25:49.512195 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:49.512218 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:49.512392 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:49.512944 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:49.512950 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:49.512964 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0810 23:25:49.513324 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:49.513339 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:49.514624 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:49.514631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:49.516484 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:49.516505 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:49.517455 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:49.517463 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:49.517682 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:49.517737 (+ 55us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 23:25:49.978463 (+460726us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 23:25:49.978746 (+ 283us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.175.126:43959: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":432,"threads_started":1}
I20260810 23:25:49.980454 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 23483
I20260810 23:25:49.981676 23581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:25:50.087262 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 23483
2026-08-10T23:25:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1242 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (446 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (444 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:25:51 dist-test-slave-rmhg krb5kdc[23656](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:25:51 dist-test-slave-rmhg krb5kdc[23656](info): set up 2 sockets
Aug 10 23:25:51 dist-test-slave-rmhg krb5kdc[23656](info): commencing operation
krb5kdc: starting...
W20260810 23:25:52.199306 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.173s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:25:52 dist-test-slave-rmhg krb5kdc[23656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:25:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:25:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
I20260810 23:25:52.442201 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:39967
--webserver_interface=127.14.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:39105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:39967
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:52.788390 23677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:52.788730 23677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:52.788846 23677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:52.796397 23677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:25:52.796525 23677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:52.796573 23677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:52.796662 23677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:25:52.796722 23677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:25:52.807164 23677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:39105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:39967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:39967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:52.808657 23677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:52.810089 23677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:52.822556 23684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:52.822613 23689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:52.822613 23686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:52.823724 23677 server_base.cc:1034] running on GCE node
I20260810 23:25:52.824641 23677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:52.826265 23677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:52.827472 23677 hybrid_clock.cc:648] HybridClock initialized: now 1786404352827418 us; error 41 us; skew 500 ppm
I20260810 23:25:52.827831 23677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:52 dist-test-slave-rmhg krb5kdc[23656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404352, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:52.835628 23677 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:25:52.837363 23677 webserver.cc:460] Webserver started at http://127.14.175.126:39681/ using document root <none> and password file <none>
I20260810 23:25:52.838109 23677 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:52.838231 23677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:52.838558 23677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:52.841815 23677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "92161fcdbfc3446d9ad8c41c18f43984"
format_stamp: "Formatted at 2026-08-10 23:25:52 on dist-test-slave-rmhg"
server_key: "1af9a9c4bb9bfe79e44f61bbd6614c0d"
server_key_iv: "15ec7e369d99cb9d8bd4d31507ac36e1"
server_key_version: "encryptionkey@0"
I20260810 23:25:52.842615 23677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "92161fcdbfc3446d9ad8c41c18f43984"
format_stamp: "Formatted at 2026-08-10 23:25:52 on dist-test-slave-rmhg"
server_key: "1af9a9c4bb9bfe79e44f61bbd6614c0d"
server_key_iv: "15ec7e369d99cb9d8bd4d31507ac36e1"
server_key_version: "encryptionkey@0"
I20260810 23:25:52.848182 23677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260810 23:25:52.851936 23700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:52.853543 23677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:25:52.853742 23677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "92161fcdbfc3446d9ad8c41c18f43984"
format_stamp: "Formatted at 2026-08-10 23:25:52 on dist-test-slave-rmhg"
server_key: "1af9a9c4bb9bfe79e44f61bbd6614c0d"
server_key_iv: "15ec7e369d99cb9d8bd4d31507ac36e1"
server_key_version: "encryptionkey@0"
I20260810 23:25:52.853951 23677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:52.880110 23677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:52.884879 23677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:52.885218 23677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:52.903040 23677 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:39967
I20260810 23:25:52.903065 23784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:39967 every 8 connection(s)
I20260810 23:25:52.904431 23677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:25:52.907960 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 23677
I20260810 23:25:52.908205 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:25:52.908819 23785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:52.909053 15037 external_mini_cluster.cc:1428] Setting key 30d383ee91b1d453ce654b91fc4b6627
I20260810 23:25:52.925452 23785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92161fcdbfc3446d9ad8c41c18f43984: Bootstrap starting.
I20260810 23:25:52.929836 23785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92161fcdbfc3446d9ad8c41c18f43984: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:52.931277 23785 log.cc:824] T 00000000000000000000000000000000 P 92161fcdbfc3446d9ad8c41c18f43984: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:25:52 dist-test-slave-rmhg krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:52.942745 23785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92161fcdbfc3446d9ad8c41c18f43984: No bootstrap required, opened a new log
I20260810 23:25:52.943938 23789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:52.912095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60498 (local address 127.14.175.126:39967)
0810 23:25:52.918163 (+ 6068us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:52.918188 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:52.918251 (+ 63us) server_negotiation.cc:407] Connection header received
0810 23:25:52.920646 (+ 2395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:52.920681 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:52.921312 (+ 631us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:52.921845 (+ 533us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:52.923910 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:52.926163 (+ 2253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:52.927472 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:52.928622 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:52.928830 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:52.933850 (+ 5020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:52.933863 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:52.933890 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:52.933963 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:52.938381 (+ 4418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:52.939463 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:52.939484 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:52.939494 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:52.939756 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:52.940247 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:52.940271 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:52.940274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:52.941076 (+ 802us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:52.941446 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:52.941975 (+ 529us) server_negotiation.cc:299] Negotiation successful
0810 23:25:52.942467 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5775,"thread_start_us":5507,"threads_started":1}
I20260810 23:25:52.953096 23785 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92161fcdbfc3446d9ad8c41c18f43984 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92161fcdbfc3446d9ad8c41c18f43984" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 39967 } }
I20260810 23:25:52.953667 23785 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92161fcdbfc3446d9ad8c41c18f43984 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:52.953780 23785 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92161fcdbfc3446d9ad8c41c18f43984 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92161fcdbfc3446d9ad8c41c18f43984, State: Initialized, Role: FOLLOWER
I20260810 23:25:52.954509 23785 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92161fcdbfc3446d9ad8c41c18f43984 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92161fcdbfc3446d9ad8c41c18f43984" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 39967 } }
I20260810 23:25:52.954819 23785 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92161fcdbfc3446d9ad8c41c18f43984 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:25:52.954980 23785 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92161fcdbfc3446d9ad8c41c18f43984 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:25:52.955188 23785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92161fcdbfc3446d9ad8c41c18f43984 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:52.958824 23785 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92161fcdbfc3446d9ad8c41c18f43984 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92161fcdbfc3446d9ad8c41c18f43984" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 39967 } }
I20260810 23:25:52.959549 23785 leader_election.cc:302] T 00000000000000000000000000000000 P 92161fcdbfc3446d9ad8c41c18f43984 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92161fcdbfc3446d9ad8c41c18f43984; no voters:
I20260810 23:25:52.960341 23785 leader_election.cc:288] T 00000000000000000000000000000000 P 92161fcdbfc3446d9ad8c41c18f43984 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:25:52.960454 23791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92161fcdbfc3446d9ad8c41c18f43984 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:25:52.962945 23785 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92161fcdbfc3446d9ad8c41c18f43984 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:25:52.967633 23791 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92161fcdbfc3446d9ad8c41c18f43984 [term 1 LEADER]: Becoming Leader. State: Replica: 92161fcdbfc3446d9ad8c41c18f43984, State: Running, Role: LEADER
I20260810 23:25:52.968641 23791 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92161fcdbfc3446d9ad8c41c18f43984 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92161fcdbfc3446d9ad8c41c18f43984" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 39967 } }
I20260810 23:25:52.972388 23792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92161fcdbfc3446d9ad8c41c18f43984 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92161fcdbfc3446d9ad8c41c18f43984" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92161fcdbfc3446d9ad8c41c18f43984" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 39967 } } }
I20260810 23:25:52.973502 23792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92161fcdbfc3446d9ad8c41c18f43984 [sys.catalog]: This master's current role is: LEADER
I20260810 23:25:52.972941 23793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92161fcdbfc3446d9ad8c41c18f43984 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92161fcdbfc3446d9ad8c41c18f43984. Latest consensus state: current_term: 1 leader_uuid: "92161fcdbfc3446d9ad8c41c18f43984" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92161fcdbfc3446d9ad8c41c18f43984" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 39967 } } }
I20260810 23:25:52.973942 23793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92161fcdbfc3446d9ad8c41c18f43984 [sys.catalog]: This master's current role is: LEADER
W20260810 23:25:52.980031 23810 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 92161fcdbfc3446d9ad8c41c18f43984: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:25:52.980175 23810 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:25:52.980309 23812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:25:52.982501 23812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:25:52.992138 23812 catalog_manager.cc:1347] Generated new cluster ID: a14280ac5e2449dfb3f1c0be67fbea52
I20260810 23:25:52.992269 23812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:25:53.014439 23812 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:25:53.015879 23812 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:25:53.021929 23812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92161fcdbfc3446d9ad8c41c18f43984: Generated new TSK 0
I20260810 23:25:53.023125 23812 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
I20260810 23:25:53.100924 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.65:0
--local_ip_for_outbound_sockets=127.14.175.65
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:39967
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:39105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:53.463050 23820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:53.463342 23820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:53.463483 23820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:53.470928 23820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:53.471045 23820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:53.471195 23820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.65
I20260810 23:25:53.481190 23820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:39105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:39967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.65
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:53.482573 23820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:53.483963 23820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:53.497610 23828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:53.497596 23830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:53.497635 23827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:53.502815 23820 server_base.cc:1034] running on GCE node
I20260810 23:25:53.503477 23820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:53.504499 23820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:53.505712 23820 hybrid_clock.cc:648] HybridClock initialized: now 1786404353505658 us; error 46 us; skew 500 ppm
I20260810 23:25:53.506043 23820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:53 dist-test-slave-rmhg krb5kdc[23656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404353, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:53.509428 23820 init.cc:377] Logged in from keytab as kudu/127.14.175.65@KRBTEST.COM (short username kudu)
I20260810 23:25:53.511088 23820 webserver.cc:460] Webserver started at http://127.14.175.65:43099/ using document root <none> and password file <none>
I20260810 23:25:53.511833 23820 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:53.511945 23820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:53.512223 23820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:53.515339 23820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-0/data/instance:
uuid: "e28107720ae24ee28e38e8d92467790f"
format_stamp: "Formatted at 2026-08-10 23:25:53 on dist-test-slave-rmhg"
server_key: "7ce2723ef0b285ec45fcbae547718b2c"
server_key_iv: "6defac6dd23d9ccbae71fda12e44789c"
server_key_version: "encryptionkey@0"
I20260810 23:25:53.516152 23820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance:
uuid: "e28107720ae24ee28e38e8d92467790f"
format_stamp: "Formatted at 2026-08-10 23:25:53 on dist-test-slave-rmhg"
server_key: "7ce2723ef0b285ec45fcbae547718b2c"
server_key_iv: "6defac6dd23d9ccbae71fda12e44789c"
server_key_version: "encryptionkey@0"
I20260810 23:25:53.521785 23820 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 23:25:53.525625 23840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:53.527026 23820 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:25:53.527176 23820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-0/wal
uuid: "e28107720ae24ee28e38e8d92467790f"
format_stamp: "Formatted at 2026-08-10 23:25:53 on dist-test-slave-rmhg"
server_key: "7ce2723ef0b285ec45fcbae547718b2c"
server_key_iv: "6defac6dd23d9ccbae71fda12e44789c"
server_key_version: "encryptionkey@0"
I20260810 23:25:53.527391 23820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:53.539077 23820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:53.543280 23820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:53.543579 23820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:53.544627 23820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:25:53.546128 23820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:25:53.546206 23820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:53.546326 23820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:25:53.546381 23820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:53.579381 23820 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.65:42561
I20260810 23:25:53.579463 24036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.65:42561 every 8 connection(s)
I20260810 23:25:53.580825 23820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
I20260810 23:25:53.590492 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 23820
I20260810 23:25:53.590750 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance
I20260810 23:25:53.591352 15037 external_mini_cluster.cc:1428] Setting key 56c85814da98afc66fd690cf6d5ba106
Aug 10 23:25:53 dist-test-slave-rmhg krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404353, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:53.602825 24040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:53.583718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:44855 (local address 127.14.175.126:39967)
0810 23:25:53.584183 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:53.584188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:53.585548 (+ 1360us) server_negotiation.cc:407] Connection header received
0810 23:25:53.587481 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:53.587487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:53.587584 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:53.587693 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:53.589528 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:53.590417 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:53.592378 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:53.592805 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:53.592925 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:53.596731 (+ 3806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:53.596737 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:53.596740 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:53.596780 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:53.599399 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:53.600244 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:53.600250 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:53.600252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:53.600400 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:53.600874 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:53.600878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:53.600881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:53.601219 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:53.601363 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:53.602185 (+ 822us) server_negotiation.cc:299] Negotiation successful
0810 23:25:53.602453 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":213,"threads_started":1}
I20260810 23:25:53.603873 24041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:53.584300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:39967 (local address 127.14.175.65:44855)
0810 23:25:53.585309 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:53.585410 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:53.587160 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:53.587991 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:53.588001 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:53.588614 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:53.589320 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:53.589344 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:53.590674 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:53.590679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:53.592187 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:53.592206 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:53.593063 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:53.593071 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:53.593378 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:53.594096 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:53.594122 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:53.596453 (+ 2331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:53.599579 (+ 3126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:53.599589 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:53.599607 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:53.600094 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:53.600517 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:53.600522 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:53.600525 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:53.600747 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:53.601356 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:53.601365 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:53.601755 (+ 390us) client_negotiation.cc:769] Sending connection context
0810 23:25:53.602025 (+ 270us) client_negotiation.cc:240] Negotiation successful
0810 23:25:53.602627 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":464,"threads_started":1}
I20260810 23:25:53.605629 24037 heartbeater.cc:347] Connected to a master server at 127.14.175.126:39967
I20260810 23:25:53.605960 24037 heartbeater.cc:464] Registering TS with master...
I20260810 23:25:53.606714 24037 heartbeater.cc:511] Master 127.14.175.126:39967 requested a full tablet report, sending...
I20260810 23:25:53.609337 23726 ts_manager.cc:194] Registered new tserver with Master: e28107720ae24ee28e38e8d92467790f (127.14.175.65:42561)
I20260810 23:25:53.611510 23726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.65@KRBTEST.COM'} at 127.14.175.65:44855
WARNING: no policy specified for kudu/127.14.175.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
I20260810 23:25:53.646432 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.66:0
--local_ip_for_outbound_sockets=127.14.175.66
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:39967
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:39105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:53.992252 24049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:53.992595 24049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:53.992754 24049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:54.000211 24049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:54.000351 24049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:54.000525 24049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.66
I20260810 23:25:54.010821 24049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:39105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:39967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.66
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:54.012272 24049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:54.013716 24049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:54.027747 24055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:54.027863 24062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:54.027768 24054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:54.028757 24049 server_base.cc:1034] running on GCE node
I20260810 23:25:54.029488 24049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:54.030542 24049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:54.031773 24049 hybrid_clock.cc:648] HybridClock initialized: now 1786404354031711 us; error 48 us; skew 500 ppm
I20260810 23:25:54.032136 24049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:54 dist-test-slave-rmhg krb5kdc[23656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404354, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:54.035439 24049 init.cc:377] Logged in from keytab as kudu/127.14.175.66@KRBTEST.COM (short username kudu)
I20260810 23:25:54.036998 24049 webserver.cc:460] Webserver started at http://127.14.175.66:39173/ using document root <none> and password file <none>
I20260810 23:25:54.037834 24049 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:54.037963 24049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:54.038267 24049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:54.041464 24049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-1/data/instance:
uuid: "f77c77c0e9cb4a11968d0dbeeb6e0075"
format_stamp: "Formatted at 2026-08-10 23:25:54 on dist-test-slave-rmhg"
server_key: "ae55ca9e20154bc48eff7cc8aed8d331"
server_key_iv: "093f3530f3de892d7bd3caabe28ea42e"
server_key_version: "encryptionkey@0"
I20260810 23:25:54.042275 24049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance:
uuid: "f77c77c0e9cb4a11968d0dbeeb6e0075"
format_stamp: "Formatted at 2026-08-10 23:25:54 on dist-test-slave-rmhg"
server_key: "ae55ca9e20154bc48eff7cc8aed8d331"
server_key_iv: "093f3530f3de892d7bd3caabe28ea42e"
server_key_version: "encryptionkey@0"
I20260810 23:25:54.048509 24049 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260810 23:25:54.052983 24069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:54.054553 24049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260810 23:25:54.054773 24049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-1/wal
uuid: "f77c77c0e9cb4a11968d0dbeeb6e0075"
format_stamp: "Formatted at 2026-08-10 23:25:54 on dist-test-slave-rmhg"
server_key: "ae55ca9e20154bc48eff7cc8aed8d331"
server_key_iv: "093f3530f3de892d7bd3caabe28ea42e"
server_key_version: "encryptionkey@0"
I20260810 23:25:54.055002 24049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:54.084260 24049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:54.088845 24049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:54.089187 24049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:54.090304 24049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:25:54.091845 24049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:25:54.091930 24049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:54.092052 24049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:25:54.092106 24049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:54.124605 24049 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.66:45535
I20260810 23:25:54.124687 24252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.66:45535 every 8 connection(s)
I20260810 23:25:54.126070 24049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
I20260810 23:25:54.132418 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 24049
I20260810 23:25:54.132622 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance
I20260810 23:25:54.133049 15037 external_mini_cluster.cc:1428] Setting key 847fe0b40a3f61eea4d556e284f2f91b
Aug 10 23:25:54 dist-test-slave-rmhg krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404354, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:54.149276 24256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:54.128751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:50575 (local address 127.14.175.126:39967)
0810 23:25:54.129233 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:54.129238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:54.130406 (+ 1168us) server_negotiation.cc:407] Connection header received
0810 23:25:54.132409 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:54.132416 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:54.132533 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:54.132659 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:54.134664 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:54.135647 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:54.137412 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:54.138019 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:54.138158 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:54.142317 (+ 4159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:54.142324 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:54.142328 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:54.142374 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:54.145307 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:54.146413 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:54.146419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:54.146421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:54.146598 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:54.147200 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:54.147205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:54.147207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:54.147542 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:54.147688 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:54.148673 (+ 985us) server_negotiation.cc:299] Negotiation successful
0810 23:25:54.148885 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":211,"threads_started":1}
I20260810 23:25:54.150727 24257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:54.129387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:39967 (local address 127.14.175.66:50575)
0810 23:25:54.130182 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:54.130251 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:54.132057 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:54.132944 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:54.132952 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:54.133598 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:54.134425 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:54.134454 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:54.135817 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:54.135822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:54.137231 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:54.137247 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:54.138256 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:54.138261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:54.138488 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:54.139249 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:54.139275 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:54.141986 (+ 2711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:54.145493 (+ 3507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:54.145505 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:54.145536 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:54.146219 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:54.146728 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:54.146733 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:54.146737 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:54.147032 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:54.147701 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:54.147711 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:54.148222 (+ 511us) client_negotiation.cc:769] Sending connection context
0810 23:25:54.148554 (+ 332us) client_negotiation.cc:240] Negotiation successful
0810 23:25:54.149187 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":300,"threads_started":1}
I20260810 23:25:54.152267 24253 heartbeater.cc:347] Connected to a master server at 127.14.175.126:39967
I20260810 23:25:54.152598 24253 heartbeater.cc:464] Registering TS with master...
I20260810 23:25:54.153419 24253 heartbeater.cc:511] Master 127.14.175.126:39967 requested a full tablet report, sending...
I20260810 23:25:54.155076 23726 ts_manager.cc:194] Registered new tserver with Master: f77c77c0e9cb4a11968d0dbeeb6e0075 (127.14.175.66:45535)
I20260810 23:25:54.156291 23726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.66@KRBTEST.COM'} at 127.14.175.66:50575
WARNING: no policy specified for kudu/127.14.175.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
I20260810 23:25:54.189786 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.67:0
--local_ip_for_outbound_sockets=127.14.175.67
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:39967
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:39105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:54.536624 24264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:54.536940 24264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:54.537086 24264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:54.544363 24264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:54.544523 24264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:54.544691 24264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.67
I20260810 23:25:54.554942 24264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:39105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:39967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.67
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:54.556339 24264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:54.557827 24264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:54.571528 24270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:54.571658 24271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:54.571658 24275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:54.572780 24264 server_base.cc:1034] running on GCE node
I20260810 23:25:54.573496 24264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:54.574568 24264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:54.575779 24264 hybrid_clock.cc:648] HybridClock initialized: now 1786404354575722 us; error 48 us; skew 500 ppm
I20260810 23:25:54.576112 24264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:54 dist-test-slave-rmhg krb5kdc[23656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404354, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:54.579389 24264 init.cc:377] Logged in from keytab as kudu/127.14.175.67@KRBTEST.COM (short username kudu)
I20260810 23:25:54.580996 24264 webserver.cc:460] Webserver started at http://127.14.175.67:42151/ using document root <none> and password file <none>
I20260810 23:25:54.581743 24264 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:54.581856 24264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:54.582183 24264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:54.585343 24264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-2/data/instance:
uuid: "fd76102c2710472d8c2d74ae95addd7b"
format_stamp: "Formatted at 2026-08-10 23:25:54 on dist-test-slave-rmhg"
server_key: "e3e0e0ed134cb40881916b569702b440"
server_key_iv: "c364bc34a3045d7b7e39382b787047f5"
server_key_version: "encryptionkey@0"
I20260810 23:25:54.586154 24264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance:
uuid: "fd76102c2710472d8c2d74ae95addd7b"
format_stamp: "Formatted at 2026-08-10 23:25:54 on dist-test-slave-rmhg"
server_key: "e3e0e0ed134cb40881916b569702b440"
server_key_iv: "c364bc34a3045d7b7e39382b787047f5"
server_key_version: "encryptionkey@0"
I20260810 23:25:54.591612 24264 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:25:54.595310 24295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:54.596710 24264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:25:54.596876 24264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-2/wal
uuid: "fd76102c2710472d8c2d74ae95addd7b"
format_stamp: "Formatted at 2026-08-10 23:25:54 on dist-test-slave-rmhg"
server_key: "e3e0e0ed134cb40881916b569702b440"
server_key_iv: "c364bc34a3045d7b7e39382b787047f5"
server_key_version: "encryptionkey@0"
I20260810 23:25:54.597085 24264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:54.608763 24264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:54.613379 24264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:54.613672 24264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:54.614708 24264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:25:54.615413 24037 heartbeater.cc:503] Master 127.14.175.126:39967 was elected leader, sending a full tablet report...
I20260810 23:25:54.616204 24264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:25:54.616282 24264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:54.616423 24264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:25:54.616477 24264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:54.647619 24264 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.67:36259
I20260810 23:25:54.647665 24472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.67:36259 every 8 connection(s)
I20260810 23:25:54.648975 24264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
I20260810 23:25:54.655664 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 24264
I20260810 23:25:54.655897 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance
I20260810 23:25:54.656467 15037 external_mini_cluster.cc:1428] Setting key c9cacac739669e22abbb417cbd289e6a
Aug 10 23:25:54 dist-test-slave-rmhg krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404354, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:25:54.671572 24477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:54.651898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:43545 (local address 127.14.175.126:39967)
0810 23:25:54.652393 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:54.652398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:54.653569 (+ 1171us) server_negotiation.cc:407] Connection header received
0810 23:25:54.655463 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:54.655468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:54.655568 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:54.655665 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:54.657821 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:54.658862 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:54.661002 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:54.661480 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:54.661591 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:54.665714 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:54.665719 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:54.665722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:54.665763 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:54.668253 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:54.669113 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:54.669119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:54.669121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:54.669301 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:54.669721 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:54.669724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:54.669726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:54.670005 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:54.670130 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:54.670999 (+ 869us) server_negotiation.cc:299] Negotiation successful
0810 23:25:54.671238 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":243,"threads_started":1}
I20260810 23:25:54.672742 24481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:54.652514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:39967 (local address 127.14.175.67:43545)
0810 23:25:54.653373 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:54.653415 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:54.655133 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:54.655978 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:54.656000 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:54.656786 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:54.657638 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:54.657666 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:54.659098 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:54.659104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:54.660811 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:54.660835 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:54.661697 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:54.661703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:54.661925 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:54.662751 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:54.662777 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:54.665445 (+ 2668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:54.668409 (+ 2964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:54.668418 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:54.668434 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:54.668972 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:54.669407 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:54.669410 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:54.669413 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:54.669629 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:54.670098 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:54.670105 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:54.670583 (+ 478us) client_negotiation.cc:769] Sending connection context
0810 23:25:54.670861 (+ 278us) client_negotiation.cc:240] Negotiation successful
0810 23:25:54.671413 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":304,"threads_started":1}
I20260810 23:25:54.674106 24475 heartbeater.cc:347] Connected to a master server at 127.14.175.126:39967
I20260810 23:25:54.674414 24475 heartbeater.cc:464] Registering TS with master...
I20260810 23:25:54.675129 24475 heartbeater.cc:511] Master 127.14.175.126:39967 requested a full tablet report, sending...
I20260810 23:25:54.676559 23726 ts_manager.cc:194] Registered new tserver with Master: fd76102c2710472d8c2d74ae95addd7b (127.14.175.67:36259)
I20260810 23:25:54.677416 23726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.67@KRBTEST.COM'} at 127.14.175.67:43545
I20260810 23:25:54.685420 15037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/test-admin.keytab.
Aug 10 23:25:54 dist-test-slave-rmhg krb5kdc[23656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:54.717464 15037 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 23:25:54.723140 24488 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:25:54.720343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47146 (local address 127.14.175.65:42561)
0810 23:25:54.720962 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:54.720971 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:54.721001 (+ 30us) server_negotiation.cc:407] Connection header received
0810 23:25:54.721192 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:54.721200 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:54.721425 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:54.721567 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:54.722711 (+ 1144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47146: BlockingRecv error: recv got EOF from 127.0.0.1:47146 (error 108)
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":275,"threads_started":1}
W20260810 23:25:54.723296 15037 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/krb5cc' of type FILE
Aug 10 23:25:54 dist-test-slave-rmhg krb5kdc[23656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:54.725250 15037 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 23:25:54 dist-test-slave-rmhg krb5kdc[23656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:54.726679 15037 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 23:25:54 dist-test-slave-rmhg krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.65@KRBTEST.COM
I20260810 23:25:54.743593 24488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:54.728509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47160 (local address 127.14.175.65:42561)
0810 23:25:54.728733 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:54.728740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:54.728915 (+ 175us) server_negotiation.cc:407] Connection header received
0810 23:25:54.729123 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:54.729129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:54.729248 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:54.729346 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:54.730451 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:54.731617 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:54.732894 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:54.733578 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:54.733703 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:54.737524 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:54.737569 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:54.737587 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:54.737639 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:54.740602 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:54.741465 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:54.741474 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:54.741476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:54.741678 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:54.742043 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:54.742046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:54.742048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:54.742455 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:54.742741 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:54.742997 (+ 256us) server_negotiation.cc:299] Negotiation successful
0810 23:25:54.743256 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260810 23:25:54.750036 24488 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:25:54.748143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47176 (local address 127.14.175.65:42561)
0810 23:25:54.748371 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:54.748377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:54.748499 (+ 122us) server_negotiation.cc:407] Connection header received
0810 23:25:54.748693 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:54.748698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:54.748775 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:54.748861 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:54.749769 (+ 908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47176: BlockingRecv error: recv got EOF from 127.0.0.1:47176 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260810 23:25:54.750156 15037 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/krb5cc' of type FILE
Aug 10 23:25:54 dist-test-slave-rmhg krb5kdc[23656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:54.751744 15037 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 23:25:54 dist-test-slave-rmhg krb5kdc[23656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:54.753057 15037 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 23:25:54 dist-test-slave-rmhg krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.65@KRBTEST.COM
I20260810 23:25:54.768679 24488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:54.754556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47180 (local address 127.14.175.65:42561)
0810 23:25:54.754789 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:54.754795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:54.755027 (+ 232us) server_negotiation.cc:407] Connection header received
0810 23:25:54.755165 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:54.755169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:54.755233 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:54.755316 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:54.756223 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:54.757059 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:54.758433 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:54.758944 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:54.759074 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:54.762824 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:54.762831 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:54.762834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:54.762879 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:54.765882 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:54.766868 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:54.766872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:54.766875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:54.767035 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:54.767428 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:54.767431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:54.767433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:54.767713 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:54.767919 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:54.768179 (+ 260us) server_negotiation.cc:299] Negotiation successful
0810 23:25:54.768367 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260810 23:25:54.773802 24488 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:25:54.771795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47190 (local address 127.14.175.65:42561)
0810 23:25:54.772018 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:54.772024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:54.772310 (+ 286us) server_negotiation.cc:407] Connection header received
0810 23:25:54.772425 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:54.772429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:54.772517 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:54.772615 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:54.773502 (+ 887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47190: BlockingRecv error: recv got EOF from 127.0.0.1:47190 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260810 23:25:54.773912 15037 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404299514076-15037-0/krb5kdc/krb5cc' of type FILE
Aug 10 23:25:54 dist-test-slave-rmhg krb5kdc[23656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:54.775502 15037 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 23:25:54 dist-test-slave-rmhg krb5kdc[23656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:54.776885 15037 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 23:25:54 dist-test-slave-rmhg krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.65@KRBTEST.COM
I20260810 23:25:54.791970 24488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:54.778504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47202 (local address 127.14.175.65:42561)
0810 23:25:54.778772 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:54.778778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:54.778866 (+ 88us) server_negotiation.cc:407] Connection header received
0810 23:25:54.779078 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:54.779083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:54.779154 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:54.779244 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:54.780231 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:54.781016 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:54.782341 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:54.782780 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:54.782886 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:54.786691 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:54.786697 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:54.786701 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:54.786746 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:54.789467 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:54.790184 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:54.790188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:54.790190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:54.790353 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:54.790655 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:54.790659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:54.790663 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:54.790943 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:54.791200 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:54.791425 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 23:25:54.791614 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260810 23:25:54.793933 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 23820
I20260810 23:25:54.806408 24488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:54.794732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47210 (local address 127.14.175.65:42561)
0810 23:25:54.794911 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:54.794916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:54.795213 (+ 297us) server_negotiation.cc:407] Connection header received
0810 23:25:54.795351 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:54.795354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:54.795416 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:54.795496 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:54.796492 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:54.797252 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:54.798431 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:54.798964 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:54.799097 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:54.801471 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:54.801477 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:54.801480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:54.801516 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:54.803898 (+ 2382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:54.804594 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:54.804598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:54.804601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:54.804745 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:54.805091 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:54.805095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:54.805096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:54.805422 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:54.805640 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:54.805947 (+ 307us) server_negotiation.cc:299] Negotiation successful
0810 23:25:54.806146 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:25:54.807019 24026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:25:54.891940 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 23820
I20260810 23:25:54.917433 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 24049
Aug 10 23:25:54 dist-test-slave-rmhg krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.66@KRBTEST.COM
I20260810 23:25:54.934355 24511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:54.918363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41986 (local address 127.14.175.66:45535)
0810 23:25:54.918927 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:54.918935 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:54.918964 (+ 29us) server_negotiation.cc:407] Connection header received
0810 23:25:54.919029 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:54.919036 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:54.919268 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:54.919406 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:54.920506 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:54.921707 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:54.923115 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:54.923695 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:54.923802 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:54.928094 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:54.928113 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:54.928131 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:54.928173 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:54.931354 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:54.932165 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:54.932175 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:54.932177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:54.932322 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:54.932666 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:54.932669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:54.932671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:54.933052 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:54.933479 (+ 427us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:54.933754 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 23:25:54.934001 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":276,"threads_started":1}
I20260810 23:25:54.935997 24244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:25:55.019660 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 24049
I20260810 23:25:55.042870 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 24264
Aug 10 23:25:55 dist-test-slave-rmhg krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.67@KRBTEST.COM
I20260810 23:25:55.060177 24515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:55.043971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38036 (local address 127.14.175.67:36259)
0810 23:25:55.044668 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:55.044676 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:55.044710 (+ 34us) server_negotiation.cc:407] Connection header received
0810 23:25:55.044772 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:55.044779 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:55.045040 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:55.045226 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:55.046405 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:55.047514 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:55.048816 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:55.049619 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:55.049738 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:55.053988 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:55.054005 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:55.054024 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:55.054068 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:55.056997 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:55.057874 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:55.057883 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:55.057885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:55.058022 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:55.058456 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:55.058508 (+ 52us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:55.058511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:55.059005 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:55.059294 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:55.059658 (+ 364us) server_negotiation.cc:299] Negotiation successful
0810 23:25:55.059884 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":338,"threads_started":1}
I20260810 23:25:55.061601 24467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:25:55.144886 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 24264
I20260810 23:25:55.164740 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 23677
I20260810 23:25:55.166097 23776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:25:55.273701 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 23677
2026-08-10T23:25:55Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4300 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:25:55 dist-test-slave-rmhg krb5kdc[24524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:25:55 dist-test-slave-rmhg krb5kdc[24524](info): set up 2 sockets
Aug 10 23:25:55 dist-test-slave-rmhg krb5kdc[24524](info): commencing operation
krb5kdc: starting...
W20260810 23:25:56.506887 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.180s user 0.005s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:25:56 dist-test-slave-rmhg krb5kdc[24524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:25:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:25:56Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.126.keytab.
Entry for principal oryx/127.14.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.126.keytab.
I20260810 23:25:56.761009 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:46435
--webserver_interface=127.14.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.126.keytab
--principal=oryx/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:36377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:46435
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:57.104004 24540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:57.104328 24540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:57.104451 24540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:57.111874 24540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:25:57.112010 24540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:57.112064 24540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:57.112154 24540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:25:57.112214 24540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:25:57.122473 24540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:36377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:46435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.126.keytab
--principal=oryx/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:46435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:57.123960 24540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:57.125402 24540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:57.137564 24547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:57.137624 24550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:57.137663 24548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:57.138721 24540 server_base.cc:1034] running on GCE node
I20260810 23:25:57.139523 24540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:57.141103 24540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:57.142340 24540 hybrid_clock.cc:648] HybridClock initialized: now 1786404357142290 us; error 43 us; skew 500 ppm
I20260810 23:25:57.142716 24540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:57 dist-test-slave-rmhg krb5kdc[24524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404357, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:57.145987 24540 init.cc:377] Logged in from keytab as oryx/127.14.175.126@KRBTEST.COM (short username oryx)
I20260810 23:25:57.147576 24540 webserver.cc:460] Webserver started at http://127.14.175.126:44067/ using document root <none> and password file <none>
I20260810 23:25:57.148286 24540 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:57.148407 24540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:57.148708 24540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:57.151831 24540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "79ba9107e6d44ecf90ec45f01d3ec450"
format_stamp: "Formatted at 2026-08-10 23:25:57 on dist-test-slave-rmhg"
server_key: "4ea36733da01801358b057382c341025"
server_key_iv: "54f6333b5afc013e154b7f9a101bf4ea"
server_key_version: "encryptionkey@0"
I20260810 23:25:57.152611 24540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "79ba9107e6d44ecf90ec45f01d3ec450"
format_stamp: "Formatted at 2026-08-10 23:25:57 on dist-test-slave-rmhg"
server_key: "4ea36733da01801358b057382c341025"
server_key_iv: "54f6333b5afc013e154b7f9a101bf4ea"
server_key_version: "encryptionkey@0"
I20260810 23:25:57.157883 24540 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260810 23:25:57.161500 24562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:57.162928 24540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260810 23:25:57.163106 24540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "79ba9107e6d44ecf90ec45f01d3ec450"
format_stamp: "Formatted at 2026-08-10 23:25:57 on dist-test-slave-rmhg"
server_key: "4ea36733da01801358b057382c341025"
server_key_iv: "54f6333b5afc013e154b7f9a101bf4ea"
server_key_version: "encryptionkey@0"
I20260810 23:25:57.163307 24540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:57.182578 24540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:57.187165 24540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:57.187453 24540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:57.206233 24540 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:46435
I20260810 23:25:57.206272 24643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:46435 every 8 connection(s)
I20260810 23:25:57.207674 24540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:25:57.211745 24644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:57.216647 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 24540
I20260810 23:25:57.216914 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:25:57.217775 15037 external_mini_cluster.cc:1428] Setting key 64894d19f02baa39729a7d12061e3a0f
I20260810 23:25:57.225143 24644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450: Bootstrap starting.
I20260810 23:25:57.229053 24644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:57.230312 24644 log.cc:824] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450: Log is configured to *not* fsync() on all Append() calls
I20260810 23:25:57.233299 24644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450: No bootstrap required, opened a new log
Aug 10 23:25:57 dist-test-slave-rmhg krb5kdc[24524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM
I20260810 23:25:57.241766 24644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } }
I20260810 23:25:57.242182 24644 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:57.242286 24644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79ba9107e6d44ecf90ec45f01d3ec450, State: Initialized, Role: FOLLOWER
I20260810 23:25:57.242919 24650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:57.220161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49702 (local address 127.14.175.126:46435)
0810 23:25:57.221072 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:57.221086 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:57.221133 (+ 47us) server_negotiation.cc:407] Connection header received
0810 23:25:57.223448 (+ 2315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:57.223477 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:57.223930 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:57.224302 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:57.226336 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:57.227840 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:57.229697 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:57.230426 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:57.230640 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:57.234697 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:57.234706 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:57.234724 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:57.234770 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:57.238154 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:57.238935 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:57.238943 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:57.238951 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:57.239163 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:57.239578 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:57.239582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:57.239584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:57.240296 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:57.240653 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:57.241127 (+ 474us) server_negotiation.cc:299] Negotiation successful
0810 23:25:57.241670 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":431,"threads_started":1}
I20260810 23:25:57.242913 24644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } }
I20260810 23:25:57.243180 24644 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:25:57.243315 24644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:25:57.243505 24644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:57.246133 24644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } }
I20260810 23:25:57.246719 24644 leader_election.cc:302] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79ba9107e6d44ecf90ec45f01d3ec450; no voters:
I20260810 23:25:57.247305 24644 leader_election.cc:288] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:25:57.247447 24652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:25:57.248123 24652 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [term 1 LEADER]: Becoming Leader. State: Replica: 79ba9107e6d44ecf90ec45f01d3ec450, State: Running, Role: LEADER
I20260810 23:25:57.248729 24652 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } }
I20260810 23:25:57.249472 24644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:25:57.251926 24653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } }
I20260810 23:25:57.252339 24653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [sys.catalog]: This master's current role is: LEADER
I20260810 23:25:57.253605 24655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79ba9107e6d44ecf90ec45f01d3ec450. Latest consensus state: current_term: 1 leader_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } }
I20260810 23:25:57.253942 24655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [sys.catalog]: This master's current role is: LEADER
I20260810 23:25:57.254349 24665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:25:57.259915 24665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:25:57.269764 24665 catalog_manager.cc:1347] Generated new cluster ID: 8d5f154f21774aaa952ac2fb0c6079bd
I20260810 23:25:57.269902 24665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:25:57.303752 24665 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:25:57.305225 24665 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:25:57.311856 24665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450: Generated new TSK 0
I20260810 23:25:57.312744 24665 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.65.keytab.
Entry for principal oryx/127.14.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.65.keytab.
I20260810 23:25:57.384574 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.65:0
--local_ip_for_outbound_sockets=127.14.175.65
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:46435
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.65.keytab
--principal=oryx/127.14.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:36377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:57.734032 24687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:57.734355 24687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:57.734499 24687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:57.742280 24687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:57.742424 24687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:57.742590 24687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.65
I20260810 23:25:57.753010 24687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:36377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.65.keytab
--principal=oryx/127.14.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:46435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.65
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:57.754665 24687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:57.756197 24687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:57.770530 24694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:57.770560 24697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:57.770601 24695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:57.771431 24687 server_base.cc:1034] running on GCE node
I20260810 23:25:57.772102 24687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:57.773262 24687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:57.774488 24687 hybrid_clock.cc:648] HybridClock initialized: now 1786404357774428 us; error 50 us; skew 500 ppm
I20260810 23:25:57.774861 24687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:57 dist-test-slave-rmhg krb5kdc[24524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404357, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:57.778378 24687 init.cc:377] Logged in from keytab as oryx/127.14.175.65@KRBTEST.COM (short username oryx)
I20260810 23:25:57.780045 24687 webserver.cc:460] Webserver started at http://127.14.175.65:42401/ using document root <none> and password file <none>
I20260810 23:25:57.780827 24687 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:57.780953 24687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:57.781335 24687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:57.784540 24687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-0/data/instance:
uuid: "4fd367b2b2444451b116d2d69f690b77"
format_stamp: "Formatted at 2026-08-10 23:25:57 on dist-test-slave-rmhg"
server_key: "26c18b4df06249abb0df4d7bded8a3a5"
server_key_iv: "dec8f7ca742a1df59e3cb6fe9a1daf04"
server_key_version: "encryptionkey@0"
I20260810 23:25:57.785490 24687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance:
uuid: "4fd367b2b2444451b116d2d69f690b77"
format_stamp: "Formatted at 2026-08-10 23:25:57 on dist-test-slave-rmhg"
server_key: "26c18b4df06249abb0df4d7bded8a3a5"
server_key_iv: "dec8f7ca742a1df59e3cb6fe9a1daf04"
server_key_version: "encryptionkey@0"
I20260810 23:25:57.791498 24687 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 23:25:57.795724 24708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:57.797425 24687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:25:57.797607 24687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-0/wal
uuid: "4fd367b2b2444451b116d2d69f690b77"
format_stamp: "Formatted at 2026-08-10 23:25:57 on dist-test-slave-rmhg"
server_key: "26c18b4df06249abb0df4d7bded8a3a5"
server_key_iv: "dec8f7ca742a1df59e3cb6fe9a1daf04"
server_key_version: "encryptionkey@0"
I20260810 23:25:57.797839 24687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:57.881678 24687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:57.886267 24687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:57.886551 24687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:57.887596 24687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:25:57.889103 24687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:25:57.889209 24687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260810 23:25:57.889339 24687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:25:57.889401 24687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:57.921973 24687 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.65:35079
I20260810 23:25:57.922024 24891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.65:35079 every 8 connection(s)
I20260810 23:25:57.923342 24687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
I20260810 23:25:57.923835 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 24687
I20260810 23:25:57.924063 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance
I20260810 23:25:57.924929 15037 external_mini_cluster.cc:1428] Setting key 0ceba167da4863819af56751f4f2898f
Aug 10 23:25:57 dist-test-slave-rmhg krb5kdc[24524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404357, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM
I20260810 23:25:57.944473 24894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:57.926240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:60559 (local address 127.14.175.126:46435)
0810 23:25:57.926751 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:57.926757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:57.927777 (+ 1020us) server_negotiation.cc:407] Connection header received
0810 23:25:57.929648 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:57.929653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:57.929735 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:57.929830 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:57.931403 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:57.932140 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:57.933927 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:57.934325 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:57.934440 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:57.938321 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:57.938327 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:57.938330 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:57.938367 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:57.940896 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:57.941933 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:57.941938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:57.941940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:57.942072 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:57.942620 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:57.942623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:57.942625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:57.942903 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:57.943014 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:57.943989 (+ 975us) server_negotiation.cc:299] Negotiation successful
0810 23:25:57.944199 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":242,"threads_started":1}
I20260810 23:25:57.946069 24896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:57.926828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.14.175.65:60559)
0810 23:25:57.927606 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:57.927664 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:57.929378 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:57.930082 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:57.930090 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:57.930628 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:57.931230 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:57.931249 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:57.932297 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:57.932301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:57.933783 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:57.933796 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:57.934560 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:57.934567 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:57.934808 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:57.935475 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:57.935503 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:57.938036 (+ 2533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:57.941093 (+ 3057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:57.941105 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:57.941126 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:57.941807 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:57.942191 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:57.942196 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:57.942199 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:57.942522 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:57.943030 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:57.943045 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:57.943555 (+ 510us) client_negotiation.cc:769] Sending connection context
0810 23:25:57.943907 (+ 352us) client_negotiation.cc:240] Negotiation successful
0810 23:25:57.944526 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":317,"threads_started":1}
I20260810 23:25:57.947822 24892 heartbeater.cc:347] Connected to a master server at 127.14.175.126:46435
I20260810 23:25:57.948206 24892 heartbeater.cc:464] Registering TS with master...
I20260810 23:25:57.949285 24892 heartbeater.cc:511] Master 127.14.175.126:46435 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.14.175.66@KRBTEST.COM; defaulting to no policy
I20260810 23:25:57.951773 24590 ts_manager.cc:194] Registered new tserver with Master: 4fd367b2b2444451b116d2d69f690b77 (127.14.175.65:35079)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.175.66@KRBTEST.COM" created.
I20260810 23:25:57.953423 24590 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.175.65@KRBTEST.COM'} at 127.14.175.65:60559
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.66.keytab.
Entry for principal oryx/127.14.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.66.keytab.
I20260810 23:25:57.980489 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.66:0
--local_ip_for_outbound_sockets=127.14.175.66
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:46435
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.66.keytab
--principal=oryx/127.14.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:36377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:58.323510 24900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:58.323822 24900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:58.323948 24900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:58.331571 24900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:58.331717 24900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:58.331863 24900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.66
I20260810 23:25:58.342278 24900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:36377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.66.keytab
--principal=oryx/127.14.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:46435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.66
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:58.343721 24900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:58.345247 24900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:58.358996 24907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:58.358942 24910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:58.358952 24908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:58.359819 24900 server_base.cc:1034] running on GCE node
I20260810 23:25:58.360487 24900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:58.361511 24900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:58.362718 24900 hybrid_clock.cc:648] HybridClock initialized: now 1786404358362655 us; error 56 us; skew 500 ppm
I20260810 23:25:58.363034 24900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:58 dist-test-slave-rmhg krb5kdc[24524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404358, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:58.366330 24900 init.cc:377] Logged in from keytab as oryx/127.14.175.66@KRBTEST.COM (short username oryx)
I20260810 23:25:58.367995 24900 webserver.cc:460] Webserver started at http://127.14.175.66:35477/ using document root <none> and password file <none>
I20260810 23:25:58.368702 24900 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:58.368808 24900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:58.369074 24900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:58.372192 24900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-1/data/instance:
uuid: "3892f3eb6ebb436d8bcd0f31647a4c6a"
format_stamp: "Formatted at 2026-08-10 23:25:58 on dist-test-slave-rmhg"
server_key: "2b658719537f2ecb4080ba5563ba8718"
server_key_iv: "7948ad53110b7a007d46e9557f5f598b"
server_key_version: "encryptionkey@0"
I20260810 23:25:58.373003 24900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance:
uuid: "3892f3eb6ebb436d8bcd0f31647a4c6a"
format_stamp: "Formatted at 2026-08-10 23:25:58 on dist-test-slave-rmhg"
server_key: "2b658719537f2ecb4080ba5563ba8718"
server_key_iv: "7948ad53110b7a007d46e9557f5f598b"
server_key_version: "encryptionkey@0"
I20260810 23:25:58.379637 24900 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 23:25:58.384301 24920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:58.385829 24900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:25:58.386009 24900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-1/wal
uuid: "3892f3eb6ebb436d8bcd0f31647a4c6a"
format_stamp: "Formatted at 2026-08-10 23:25:58 on dist-test-slave-rmhg"
server_key: "2b658719537f2ecb4080ba5563ba8718"
server_key_iv: "7948ad53110b7a007d46e9557f5f598b"
server_key_version: "encryptionkey@0"
I20260810 23:25:58.386216 24900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:58.401394 24900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:58.406033 24900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:58.406323 24900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:58.407316 24900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:25:58.408721 24900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:25:58.408804 24900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:58.408933 24900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:25:58.408993 24900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:58.439327 24900 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.66:36553
I20260810 23:25:58.439368 25092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.66:36553 every 8 connection(s)
I20260810 23:25:58.440683 24900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
I20260810 23:25:58.445792 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 24900
I20260810 23:25:58.445979 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance
I20260810 23:25:58.446415 15037 external_mini_cluster.cc:1428] Setting key 014fad33795504e16aaa907f4990ad32
Aug 10 23:25:58 dist-test-slave-rmhg krb5kdc[24524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404358, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM
I20260810 23:25:58.462533 24894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:58.443416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:58237 (local address 127.14.175.126:46435)
0810 23:25:58.443648 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:58.443654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:58.444891 (+ 1237us) server_negotiation.cc:407] Connection header received
0810 23:25:58.446980 (+ 2089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:58.446986 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:58.447086 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:58.447186 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:58.448856 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:58.449682 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:58.451444 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:58.451885 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:58.452005 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:58.455966 (+ 3961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:58.455972 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:58.455976 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:58.456041 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:58.459160 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:58.460022 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:58.460028 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:58.460031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:58.460236 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:58.460668 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:58.460673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:58.460676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:58.461023 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:58.461198 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:58.461934 (+ 736us) server_negotiation.cc:299] Negotiation successful
0810 23:25:58.462195 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 23:25:58.463483 25098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:58.443933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.14.175.66:58237)
0810 23:25:58.444706 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:58.444748 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:58.446619 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:58.447466 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:58.447475 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:58.448046 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:58.448672 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:58.448691 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:58.449861 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:58.449865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:58.451290 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:58.451303 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:58.452133 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:58.452141 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:58.452399 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:58.453082 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:58.453110 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:58.455657 (+ 2547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:58.459343 (+ 3686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:58.459353 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:58.459370 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:58.459871 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:58.460344 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:58.460347 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:58.460350 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:58.460560 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:58.461133 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:58.461140 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:58.461556 (+ 416us) client_negotiation.cc:769] Sending connection context
0810 23:25:58.461801 (+ 245us) client_negotiation.cc:240] Negotiation successful
0810 23:25:58.462261 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":273,"threads_started":1}
I20260810 23:25:58.464983 25093 heartbeater.cc:347] Connected to a master server at 127.14.175.126:46435
I20260810 23:25:58.465341 25093 heartbeater.cc:464] Registering TS with master...
I20260810 23:25:58.466145 25093 heartbeater.cc:511] Master 127.14.175.126:46435 requested a full tablet report, sending...
I20260810 23:25:58.468084 24590 ts_manager.cc:194] Registered new tserver with Master: 3892f3eb6ebb436d8bcd0f31647a4c6a (127.14.175.66:36553)
I20260810 23:25:58.469305 24590 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.175.66@KRBTEST.COM'} at 127.14.175.66:58237
WARNING: no policy specified for oryx/127.14.175.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.175.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.67.keytab.
Entry for principal oryx/127.14.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.67.keytab.
I20260810 23:25:58.502260 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.67:0
--local_ip_for_outbound_sockets=127.14.175.67
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:46435
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.67.keytab
--principal=oryx/127.14.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:36377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:58.851145 25102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:58.851465 25102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:58.851616 25102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:58.859225 25102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:58.859373 25102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:58.859530 25102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.67
I20260810 23:25:58.870052 25102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:36377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.67.keytab
--principal=oryx/127.14.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:46435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.67
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:25:58.871450 25102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:58.872915 25102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:58.886796 25109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:58.886881 25113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:58.887296 25108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:58.887738 25102 server_base.cc:1034] running on GCE node
I20260810 23:25:58.888382 25102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:58.889395 25102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:58.890651 25102 hybrid_clock.cc:648] HybridClock initialized: now 1786404358890555 us; error 75 us; skew 500 ppm
I20260810 23:25:58.891059 25102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:58 dist-test-slave-rmhg krb5kdc[24524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404358, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:58.894378 25102 init.cc:377] Logged in from keytab as oryx/127.14.175.67@KRBTEST.COM (short username oryx)
I20260810 23:25:58.895905 25102 webserver.cc:460] Webserver started at http://127.14.175.67:37941/ using document root <none> and password file <none>
I20260810 23:25:58.896600 25102 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:58.896708 25102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:58.896975 25102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:58.900089 25102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-2/data/instance:
uuid: "28dabb1e621d4aedbd18cb5ff0091cea"
format_stamp: "Formatted at 2026-08-10 23:25:58 on dist-test-slave-rmhg"
server_key: "d8581afb8589786664aef806ef5cfe52"
server_key_iv: "4d2b2e247c1936519c1602e2cc870be3"
server_key_version: "encryptionkey@0"
I20260810 23:25:58.900882 25102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance:
uuid: "28dabb1e621d4aedbd18cb5ff0091cea"
format_stamp: "Formatted at 2026-08-10 23:25:58 on dist-test-slave-rmhg"
server_key: "d8581afb8589786664aef806ef5cfe52"
server_key_iv: "4d2b2e247c1936519c1602e2cc870be3"
server_key_version: "encryptionkey@0"
I20260810 23:25:58.906491 25102 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 23:25:58.910298 25122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:58.911820 25102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:25:58.912007 25102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-2/wal
uuid: "28dabb1e621d4aedbd18cb5ff0091cea"
format_stamp: "Formatted at 2026-08-10 23:25:58 on dist-test-slave-rmhg"
server_key: "d8581afb8589786664aef806ef5cfe52"
server_key_iv: "4d2b2e247c1936519c1602e2cc870be3"
server_key_version: "encryptionkey@0"
I20260810 23:25:58.912235 25102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:58.924762 25102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:58.929221 25102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:58.929512 25102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:58.930523 25102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:25:58.931928 25102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:25:58.932011 25102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:58.932137 25102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:25:58.932191 25102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:58.956758 24892 heartbeater.cc:503] Master 127.14.175.126:46435 was elected leader, sending a full tablet report...
I20260810 23:25:58.968008 25102 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.67:44665
I20260810 23:25:58.968055 25304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.67:44665 every 8 connection(s)
I20260810 23:25:58.969431 25102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
I20260810 23:25:58.978341 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 25102
I20260810 23:25:58.978530 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance
I20260810 23:25:58.978971 15037 external_mini_cluster.cc:1428] Setting key f27230d1afa3524c4e84d22cc576d478
Aug 10 23:25:58 dist-test-slave-rmhg krb5kdc[24524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404358, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM
I20260810 23:25:58.992957 25309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:58.974088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:36357 (local address 127.14.175.126:46435)
0810 23:25:58.974633 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:58.974637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:58.975478 (+ 841us) server_negotiation.cc:407] Connection header received
0810 23:25:58.977486 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:58.977491 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:58.977586 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:58.977684 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:58.979486 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:58.980533 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:58.982612 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:58.983012 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:58.983127 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:58.986781 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:58.986786 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:58.986789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:58.986826 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:58.989581 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:58.990464 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:58.990468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:58.990470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:58.990609 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:58.991052 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:58.991056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:58.991058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:58.991382 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:58.991529 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:58.992317 (+ 788us) server_negotiation.cc:299] Negotiation successful
0810 23:25:58.992566 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":271,"threads_started":1}
I20260810 23:25:58.994048 25311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:58.974523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.14.175.67:36357)
0810 23:25:58.975293 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:58.975351 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:58.977123 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:58.977970 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:58.977980 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:58.978618 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:58.979293 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:58.979314 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:58.980694 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:58.980698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:58.982461 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:58.982476 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:58.983247 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:58.983255 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:58.983491 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:58.984175 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:58.984200 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:58.986537 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:58.989742 (+ 3205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:58.989753 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:58.989774 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:58.990329 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:58.990721 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:58.990726 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:58.990730 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:58.990927 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:58.991510 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:58.991519 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:58.991906 (+ 387us) client_negotiation.cc:769] Sending connection context
0810 23:25:58.992180 (+ 274us) client_negotiation.cc:240] Negotiation successful
0810 23:25:58.992761 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":273,"threads_started":1}
I20260810 23:25:58.995528 25305 heartbeater.cc:347] Connected to a master server at 127.14.175.126:46435
I20260810 23:25:58.995926 25305 heartbeater.cc:464] Registering TS with master...
I20260810 23:25:58.997056 25305 heartbeater.cc:511] Master 127.14.175.126:46435 requested a full tablet report, sending...
I20260810 23:25:58.998678 24590 ts_manager.cc:194] Registered new tserver with Master: 28dabb1e621d4aedbd18cb5ff0091cea (127.14.175.67:44665)
I20260810 23:25:58.999517 24590 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.175.67@KRBTEST.COM'} at 127.14.175.67:36357
I20260810 23:25:59.007562 15037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.14.175.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.175.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.175.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.125.keytab.
Entry for principal oryx/127.14.175.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.125.keytab.
I20260810 23:25:59.065945 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.125:32771
--webserver_interface=127.14.175.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.125.keytab
--principal=oryx/127.14.175.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:36377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:46435,127.14.175.125:32771
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:25:59.418176 25316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:25:59.418624 25316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:25:59.418776 25316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:25:59.430037 25316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:25:59.430191 25316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:25:59.430274 25316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:25:59.430358 25316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:25:59.430423 25316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:25:59.442459 25316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:36377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:46435,127.14.175.125:32771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.125.keytab
--principal=oryx/127.14.175.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.125:32771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.175.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
W20260810 23:25:59.442927 25316 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.14.175.126:46435,127.14.175.125:32771'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 23:25:59.444298 25316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:25:59.446141 25316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:59.458168 25324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:59.458220 25326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:59.458138 25323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:59.459028 25316 server_base.cc:1034] running on GCE node
I20260810 23:25:59.459805 25316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:59.461237 25316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:59.462452 25316 hybrid_clock.cc:648] HybridClock initialized: now 1786404359462381 us; error 45 us; skew 500 ppm
I20260810 23:25:59.462800 25316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:59 dist-test-slave-rmhg krb5kdc[24524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404359, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:59.466042 25316 init.cc:377] Logged in from keytab as oryx/127.14.175.125@KRBTEST.COM (short username oryx)
I20260810 23:25:59.467667 25316 webserver.cc:460] Webserver started at http://127.14.175.125:39513/ using document root <none> and password file <none>
I20260810 23:25:59.468410 25316 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:59.468565 25316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:59.468891 25316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:25:59.472316 25093 heartbeater.cc:503] Master 127.14.175.126:46435 was elected leader, sending a full tablet report...
I20260810 23:25:59.472138 25316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-1/data/instance:
uuid: "c8e7587e7a104723916c49843b278e0f"
format_stamp: "Formatted at 2026-08-10 23:25:59 on dist-test-slave-rmhg"
server_key: "08e35454cddc31350ac840bccb06b32f"
server_key_iv: "12180c715faca36beaa446991ed044d6"
server_key_version: "encryptionkey@0"
I20260810 23:25:59.473083 25316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-1/wal/instance:
uuid: "c8e7587e7a104723916c49843b278e0f"
format_stamp: "Formatted at 2026-08-10 23:25:59 on dist-test-slave-rmhg"
server_key: "08e35454cddc31350ac840bccb06b32f"
server_key_iv: "12180c715faca36beaa446991ed044d6"
server_key_version: "encryptionkey@0"
I20260810 23:25:59.479034 25316 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 23:25:59.483184 25340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:59.484886 25316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:25:59.485091 25316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-1/wal
uuid: "c8e7587e7a104723916c49843b278e0f"
format_stamp: "Formatted at 2026-08-10 23:25:59 on dist-test-slave-rmhg"
server_key: "08e35454cddc31350ac840bccb06b32f"
server_key_iv: "12180c715faca36beaa446991ed044d6"
server_key_version: "encryptionkey@0"
I20260810 23:25:59.485402 25316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:59.508365 25316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:59.512856 25316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:25:59.513127 25316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:59.531111 25316 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.125:32771
I20260810 23:25:59.531131 25427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.125:32771 every 8 connection(s)
I20260810 23:25:59.532594 25316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-1/data/info.pb
I20260810 23:25:59.536535 25428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:25:59.542026 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 25316
I20260810 23:25:59.543883 25428 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:25:59 dist-test-slave-rmhg krb5kdc[24524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404359, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.125@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM
Aug 10 23:25:59 dist-test-slave-rmhg krb5kdc[24524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.175.125@KRBTEST.COM
I20260810 23:25:59.564582 25436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:59.547783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49710 (local address 127.14.175.126:46435)
0810 23:25:59.549252 (+ 1469us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:59.549257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:59.549275 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:25:59.549354 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:59.549358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:59.549475 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:59.549583 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:59.551079 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:59.551847 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:59.553482 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:59.553898 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:59.553997 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:59.557996 (+ 3999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:59.558005 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:59.558010 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:59.558065 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:59.560914 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:59.561758 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:59.561764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:59.561768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:59.561890 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:59.562329 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:59.562333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:59.562335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:59.562702 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:59.562853 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:59.563742 (+ 889us) server_negotiation.cc:299] Negotiation successful
0810 23:25:59.563994 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1313,"thread_start_us":1211,"threads_started":1}
I20260810 23:25:59.565486 25435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:59.547705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.0.0.1:49710)
0810 23:25:59.548451 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:59.548508 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:59.548664 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:59.549690 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:59.549697 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:59.550344 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:59.550896 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:59.550919 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:59.551991 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:59.551997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:59.553334 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:59.553346 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:59.554134 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:59.554142 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:59.554329 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:59.555078 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:59.555102 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:59.557756 (+ 2654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:59.561053 (+ 3297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:59.561069 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:59.561089 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:59.561638 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:59.561997 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:59.562002 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:59.562007 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:59.562208 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:59.562835 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:59.562844 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:59.563294 (+ 450us) client_negotiation.cc:769] Sending connection context
0810 23:25:59.563588 (+ 294us) client_negotiation.cc:240] Negotiation successful
0810 23:25:59.564185 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":6,"spinlock_wait_cycles":15744,"thread_start_us":302,"threads_started":1}
I20260810 23:25:59.565514 25434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:59.544294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43190 (local address 127.14.175.125:32771)
0810 23:25:59.545515 (+ 1221us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:59.545532 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:59.545592 (+ 60us) server_negotiation.cc:407] Connection header received
0810 23:25:59.547836 (+ 2244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:59.547867 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:59.548451 (+ 584us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:59.548818 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:59.550702 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:59.552095 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:59.553419 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:59.553961 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:59.554167 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:59.558560 (+ 4393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:59.558568 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:59.558584 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:59.558619 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:59.561424 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:59.562121 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:59.562129 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:59.562131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:59.562262 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:59.562648 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:59.562652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:59.562654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:59.563258 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:59.563581 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:59.563923 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 23:25:59.564329 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":192,"server-negotiator.queue_time_us":952,"thread_start_us":698,"threads_started":1}
I20260810 23:25:59.568480 25428 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:25:59 dist-test-slave-rmhg krb5kdc[24524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404359, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.125@KRBTEST.COM for oryx/127.14.175.125@KRBTEST.COM
I20260810 23:25:59.580674 25435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:59.569379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:32771 (local address 127.0.0.1:43200)
0810 23:25:59.569538 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:59.569557 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:59.569643 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:59.569876 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:59.569879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:59.570222 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:59.570352 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:59.570359 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:59.571270 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:59.571274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:59.572134 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:59.572144 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:59.572743 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:59.572747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:59.572899 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:59.573460 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:59.573484 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:59.575544 (+ 2060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:59.578460 (+ 2916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:59.578464 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:59.578467 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:59.578892 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:59.579233 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:59.579237 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:59.579239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:59.579370 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:59.579878 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:59.579882 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:59.580029 (+ 147us) client_negotiation.cc:769] Sending connection context
0810 23:25:59.580164 (+ 135us) client_negotiation.cc:240] Negotiation successful
0810 23:25:59.580410 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 23:25:59.580801 25434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:59.569438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43200 (local address 127.14.175.125:32771)
0810 23:25:59.569583 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:59.569588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:59.569604 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:25:59.569725 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:59.569728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:59.569781 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:59.569854 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:59.570457 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:59.571154 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:59.572251 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:59.572624 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:59.572714 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:59.575744 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:59.575749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:59.575753 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:59.575783 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:59.578313 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:59.578996 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:59.579000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:59.579002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:59.579135 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:59.579455 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:59.579459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:59.579461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:59.579742 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:59.579850 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:59.580228 (+ 378us) server_negotiation.cc:299] Negotiation successful
0810 23:25:59.580536 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":119,"server-negotiator.queue_time_us":45}
I20260810 23:25:59.585819 25428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f: Bootstrap starting.
I20260810 23:25:59.589044 25428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f: Neither blocks nor log segments found. Creating new log.
I20260810 23:25:59.590191 25428 log.cc:824] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f: Log is configured to *not* fsync() on all Append() calls
I20260810 23:25:59.593041 25428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f: No bootstrap required, opened a new log
I20260810 23:25:59.600250 25428 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } }
I20260810 23:25:59.600634 25428 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:25:59.600735 25428 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8e7587e7a104723916c49843b278e0f, State: Initialized, Role: FOLLOWER
I20260810 23:25:59.601390 25428 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } }
I20260810 23:25:59.602319 25439 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } } }
I20260810 23:25:59.602665 25439 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:25:59.603333 25428 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:25:59.610670 25439 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:25:59.611065 25439 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } }
W20260810 23:25:59.612167 25457 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:25:59.612311 25457 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:25:59.612474 25439 leader_election.cc:288] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79ba9107e6d44ecf90ec45f01d3ec450 (127.14.175.126:46435)
I20260810 23:25:59.613408 24604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8e7587e7a104723916c49843b278e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" is_pre_election: true
I20260810 23:25:59.614595 25344 leader_election.cc:302] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: c8e7587e7a104723916c49843b278e0f; no voters: 79ba9107e6d44ecf90ec45f01d3ec450
I20260810 23:25:59.615065 25439 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:25:59.616048 25316 master_runner.cc:428] Detected that this master c8e7587e7a104723916c49843b278e0f is joining an existing cluster
I20260810 23:25:59.616127 25316 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.175.125:32771
I20260810 23:25:59.642535 25434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:59.620893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43212 (local address 127.14.175.125:32771)
0810 23:25:59.621054 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:59.621060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:59.629851 (+ 8791us) server_negotiation.cc:407] Connection header received
0810 23:25:59.629986 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:59.629991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:59.630054 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:59.630150 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:59.631528 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:59.632490 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:59.634334 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:59.634886 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:59.635003 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:59.636839 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:59.636845 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:59.636849 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:59.636877 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:59.639613 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:59.640243 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:59.640249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:59.640253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:59.640441 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:59.640875 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:59.640880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:59.640882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:59.641366 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:59.641503 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:59.641952 (+ 449us) server_negotiation.cc:299] Negotiation successful
0810 23:25:59.642181 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:25:59.643273 25436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:59.620339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49722 (local address 127.14.175.126:46435)
0810 23:25:59.620552 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:59.620559 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:59.627580 (+ 7021us) server_negotiation.cc:407] Connection header received
0810 23:25:59.627763 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:59.627768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:59.627876 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:59.627984 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:59.629735 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:59.630856 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:59.633018 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:59.633744 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:59.633855 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:59.636700 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:59.636708 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:59.636713 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:59.636756 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:59.640207 (+ 3451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:59.640848 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:59.640854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:59.640857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:59.641024 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:59.641518 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:59.641523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:59.641527 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:59.641918 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:59.642042 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:59.642694 (+ 652us) server_negotiation.cc:299] Negotiation successful
0810 23:25:59.642944 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 23:25:59.643535 25469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:59.621392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:32771 (local address 127.0.0.1:43212)
0810 23:25:59.629675 (+ 8283us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:59.629701 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:59.629894 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:59.630295 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:59.630299 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:59.630978 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:59.631365 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:59.631377 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:59.632722 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:59.632727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:59.634197 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:59.634210 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:59.635241 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:59.635245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:59.635356 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:59.635858 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:59.635874 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:59.636686 (+ 812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:59.639735 (+ 3049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:59.639741 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:59.639745 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:59.640140 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:59.640554 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:59.640560 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:59.640563 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:59.640742 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:59.641547 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:59.641551 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:59.641705 (+ 154us) client_negotiation.cc:769] Sending connection context
0810 23:25:59.642812 (+ 1107us) client_negotiation.cc:240] Negotiation successful
0810 23:25:59.643107 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8151,"thread_start_us":7936,"threads_started":1}
I20260810 23:25:59.644764 25468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:59.620238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.0.0.1:49722)
0810 23:25:59.627406 (+ 7168us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:59.627432 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:59.627633 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:59.628197 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:59.628202 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:59.628927 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:59.629479 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:59.629496 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:59.631039 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:59.631045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:59.632817 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:59.632832 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:59.634194 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:59.634201 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:59.634393 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:59.635175 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:59.635199 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:59.636448 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:59.640320 (+ 3872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:59.640325 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:59.640328 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:59.640725 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:59.641179 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:59.641184 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:59.641188 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:59.641378 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:59.642142 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:59.642147 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:59.642354 (+ 207us) client_negotiation.cc:769] Sending connection context
0810 23:25:59.643525 (+ 1171us) client_negotiation.cc:240] Negotiation successful
0810 23:25:59.643894 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6992,"spinlock_wait_cycles":4992,"thread_start_us":6836,"threads_started":1}
WARNING: no policy specified for oryx/127.14.175.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.175.124@KRBTEST.COM" created.
Aug 10 23:25:59 dist-test-slave-rmhg krb5kdc[24524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404357, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.126@KRBTEST.COM for oryx/127.14.175.125@KRBTEST.COM
I20260810 23:25:59.670370 25434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:59.652492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42208 (local address 127.14.175.125:32771)
0810 23:25:59.652703 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:59.652709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:59.653662 (+ 953us) server_negotiation.cc:407] Connection header received
0810 23:25:59.654151 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:59.654158 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:59.654275 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:59.654349 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:59.656246 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:59.657412 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:59.659469 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:59.660021 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:59.660145 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:59.664016 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:59.664023 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:59.664028 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:59.664068 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:59.666962 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:59.667974 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:59.667979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:59.667981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:59.668159 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:59.668607 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:59.668611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:59.668614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:59.668925 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:59.669064 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:59.669704 (+ 640us) server_negotiation.cc:299] Negotiation successful
0810 23:25:59.669974 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 23:25:59.670418 25472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:59.652555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:32771 (local address 127.0.0.1:42208)
0810 23:25:59.653375 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:59.653434 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:59.653850 (+ 416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:59.654692 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:59.654700 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:59.655689 (+ 989us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:59.656029 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:59.656046 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:59.657598 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:59.657604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:59.659305 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:59.659324 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:59.660204 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:59.660211 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:59.660441 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:59.661302 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:59.661323 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:59.663797 (+ 2474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:59.667147 (+ 3350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:59.667169 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:59.667178 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:59.667840 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:59.668286 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:59.668291 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:59.668295 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:59.668494 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:59.669071 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:59.669096 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:59.669391 (+ 295us) client_negotiation.cc:769] Sending connection context
0810 23:25:59.669596 (+ 205us) client_negotiation.cc:240] Negotiation successful
0810 23:25:59.669988 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":1204,"threads_started":1}
I20260810 23:25:59.673069 24590 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.14.175.125:32771: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c8e7587e7a104723916c49843b278e0f"
member_type: NON_VOTER
last_known_addr {
host: "127.14.175.125"
port: 32771
}
attrs {
promote: true
}
}
dest_uuid: "79ba9107e6d44ecf90ec45f01d3ec450"
cas_config_opid_index: -1
I20260810 23:25:59.674537 24590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: NON_VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: true } }
I20260810 23:25:59.679157 25392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:25:59.681808 25477 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c8e7587e7a104723916c49843b278e0f (127.14.175.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: NON_VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: true } } }
I20260810 23:25:59.683075 25392 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 1 FOLLOWER]: Refusing update from remote peer 79ba9107e6d44ecf90ec45f01d3ec450: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 23:25:59.684329 25478 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: NON_VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:25:59.697772 25477 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.14.175.125:32771
I20260810 23:25:59.698954 25316 master.cc:537] Master@127.14.175.125:32771 shutting down...
I20260810 23:25:59.703332 25478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: NON_VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: true } } }
I20260810 23:25:59.703738 25478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [sys.catalog]: This master's current role is: LEADER
I20260810 23:25:59.707993 25392 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c8e7587e7a104723916c49843b278e0f (127.14.175.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: NON_VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: true } } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.175.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.124.keytab.
Entry for principal oryx/127.14.175.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.124.keytab.
I20260810 23:25:59.713900 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.124:42991
--webserver_interface=127.14.175.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.124.keytab
--principal=oryx/127.14.175.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:36377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:46435,127.14.175.125:32771,127.14.175.124:42991
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:25:59.735954 25492 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: NON_VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: true } } }
I20260810 23:25:59.737030 25482 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: NON_VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: true } } }
I20260810 23:25:59.737372 25482 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [sys.catalog]: This master's current role is: LEARNER
I20260810 23:25:59.737983 25439 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79ba9107e6d44ecf90ec45f01d3ec450. Latest consensus state: current_term: 1 leader_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: NON_VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: true } } }
I20260810 23:25:59.738209 25439 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [sys.catalog]: This master's current role is: LEARNER
I20260810 23:25:59.739395 25316 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 1 LEARNER]: Raft consensus shutting down.
I20260810 23:25:59.739873 25316 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 1 LEARNER]: Raft consensus is shut down!
I20260810 23:25:59.736431 25492 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [sys.catalog]: This master's current role is: LEARNER
I20260810 23:25:59.740162 25316 tablet_replica.cc:330] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f: stopping tablet replica
I20260810 23:25:59.748413 25316 master.cc:552] Master@127.14.175.125:32771 shutdown complete.
I20260810 23:25:59.748580 25316 master_runner.cc:305] Clearing existing system tablet
I20260810 23:25:59.749073 25316 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:59.749203 25316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:59.752635 25316 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260810 23:25:59.754590 25495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:59.755072 25316 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:59.755595 25316 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260810 23:25:59.755739 25316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-1/wal
uuid: "c8e7587e7a104723916c49843b278e0f"
format_stamp: "Formatted at 2026-08-10 23:25:59 on dist-test-slave-rmhg"
server_key: "08e35454cddc31350ac840bccb06b32f"
server_key_iv: "12180c715faca36beaa446991ed044d6"
server_key_version: "encryptionkey@0"
I20260810 23:25:59.761050 25316 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:25:59.768800 25316 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 23:25:59.769037 25316 log.cc:1196] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 23:25:59.769616 25316 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f: Deleting consensus metadata
I20260810 23:25:59.769942 25316 master_runner.cc:315] Copying system tablet from 127.14.175.126:46435
I20260810 23:25:59.775549 25316 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.175.126:46435
I20260810 23:25:59.793275 25436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:59.776855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36734 (local address 127.14.175.126:46435)
0810 23:25:59.777092 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0810 23:25:59.777098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:25:59.777700 (+ 602us) server_negotiation.cc:407] Connection header received
0810 23:25:59.777922 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:25:59.777929 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:25:59.778023 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:25:59.778147 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:25:59.779364 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:59.780552 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:59.782562 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:25:59.783242 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:25:59.783389 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:59.786078 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:25:59.786086 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:25:59.786091 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:25:59.786152 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:25:59.789882 (+ 3730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:59.790876 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:59.790883 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:59.790887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:59.791063 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:25:59.791520 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:25:59.791526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:25:59.791529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:25:59.791925 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:25:59.792103 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0810 23:25:59.792575 (+ 472us) server_negotiation.cc:299] Negotiation successful
0810 23:25:59.792840 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 23:25:59.793304 25506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:25:59.776782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.0.0.1:36734)
0810 23:25:59.777545 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0810 23:25:59.777572 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 23:25:59.777753 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:25:59.778215 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:25:59.778220 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:25:59.778770 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:25:59.779152 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:59.779172 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:59.780756 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:59.780766 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:59.782330 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:25:59.782351 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:25:59.783449 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:25:59.783455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:25:59.783659 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:25:59.784445 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:25:59.784487 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:25:59.785796 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:25:59.790076 (+ 4280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:59.790083 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:59.790087 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:59.790710 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:59.791193 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:25:59.791198 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:25:59.791201 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:25:59.791388 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:25:59.792068 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:25:59.792073 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:25:59.792266 (+ 193us) client_negotiation.cc:769] Sending connection context
0810 23:25:59.792418 (+ 152us) client_negotiation.cc:240] Negotiation successful
0810 23:25:59.792769 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":462,"threads_started":1}
I20260810 23:25:59.794889 24620 tablet_copy_service.cc:140] P 79ba9107e6d44ecf90ec45f01d3ec450: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c8e7587e7a104723916c49843b278e0f ({username='oryx', principal='oryx/127.14.175.125@KRBTEST.COM'} at 127.0.0.1:36734)
I20260810 23:25:59.795342 24620 tablet_copy_service.cc:161] P 79ba9107e6d44ecf90ec45f01d3ec450: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c8e7587e7a104723916c49843b278e0f at {username='oryx', principal='oryx/127.14.175.125@KRBTEST.COM'} at 127.0.0.1:36734: session id = c8e7587e7a104723916c49843b278e0f-00000000000000000000000000000000
I20260810 23:25:59.800621 24620 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 23:25:59.803856 25316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 23:25:59.811444 25316 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f: tablet copy: Starting download of 0 data blocks...
I20260810 23:25:59.811993 25316 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f: tablet copy: Starting download of 1 WAL segments...
I20260810 23:25:59.816689 25316 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 23:25:59.822049 24620 tablet_copy_service.cc:342] P 79ba9107e6d44ecf90ec45f01d3ec450: Request end of tablet copy session c8e7587e7a104723916c49843b278e0f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.175.125@KRBTEST.COM'} at 127.0.0.1:36734
I20260810 23:25:59.822346 24620 tablet_copy_service.cc:433] P 79ba9107e6d44ecf90ec45f01d3ec450: ending tablet copy session c8e7587e7a104723916c49843b278e0f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c8e7587e7a104723916c49843b278e0f
I20260810 23:25:59.834084 25316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:25:59.836635 25508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:25:59.838690 25509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:59.849432 25316 server_base.cc:1034] running on GCE node
W20260810 23:25:59.849392 25512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:25:59.850085 25316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:25:59.850814 25316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:25:59.852027 25316 hybrid_clock.cc:648] HybridClock initialized: now 1786404359851974 us; error 37 us; skew 500 ppm
I20260810 23:25:59.852267 25316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:25:59 dist-test-slave-rmhg krb5kdc[24524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404359, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:25:59.854255 25316 init.cc:377] Logged in from keytab as oryx/127.14.175.125@KRBTEST.COM (short username oryx)
I20260810 23:25:59.855544 25316 webserver.cc:460] Webserver started at http://127.14.175.125:36145/ using document root <none> and password file <none>
I20260810 23:25:59.855875 25316 fs_manager.cc:359] Metadata directory not provided
I20260810 23:25:59.855952 25316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:25:59.858471 25316 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260810 23:25:59.860229 25520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:25:59.860671 25316 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:25:59.860813 25316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-1/wal
uuid: "c8e7587e7a104723916c49843b278e0f"
format_stamp: "Formatted at 2026-08-10 23:25:59 on dist-test-slave-rmhg"
server_key: "08e35454cddc31350ac840bccb06b32f"
server_key_iv: "12180c715faca36beaa446991ed044d6"
server_key_version: "encryptionkey@0"
I20260810 23:25:59.860978 25316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:25:59.869424 25316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:25:59.872669 25316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:25:59.891561 25316 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.125:32771
I20260810 23:25:59.891561 25607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.125:32771 every 8 connection(s)
I20260810 23:25:59.892474 25316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-1/data/info.pb
I20260810 23:25:59.898447 25608 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:25:59.902796 25608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f: Bootstrap starting.
I20260810 23:25:59.915290 25608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:25:59.915872 25608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f: Bootstrap complete.
I20260810 23:25:59.916870 25608 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: NON_VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: true } }
I20260810 23:25:59.917136 25608 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8e7587e7a104723916c49843b278e0f, State: Initialized, Role: LEARNER
I20260810 23:25:59.917439 25608 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: NON_VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: true } }
I20260810 23:25:59.918283 25615 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: NON_VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: true } } }
I20260810 23:25:59.918674 25608 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:25:59.918871 25615 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [sys.catalog]: This master's current role is: LEARNER
I20260810 23:25:59.936381 25634 catalog_manager.cc:1259] Loaded cluster ID: 8d5f154f21774aaa952ac2fb0c6079bd
I20260810 23:25:59.936664 25634 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f: loading cluster ID for follower catalog manager: success
I20260810 23:25:59.940448 25634 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f: acquiring CA information for follower catalog manager: success
I20260810 23:25:59.943368 25634 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:26:00.002470 25305 heartbeater.cc:503] Master 127.14.175.126:46435 was elected leader, sending a full tablet report...
W20260810 23:26:00.126895 25489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:26:00.127235 25489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:26:00.127379 25489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:26:00.135742 25489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:26:00.135918 25489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:26:00.136008 25489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:26:00.136094 25489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:26:00.136164 25489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:26:00.147069 25489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:36377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:46435,127.14.175.125:32771,127.14.175.124:42991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.124.keytab
--principal=oryx/127.14.175.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.124:42991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.175.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:26:00.148630 25489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:26:00.150202 25489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:26:00.163280 25644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:00.163286 25643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:00.163308 25647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:26:00.164202 25489 server_base.cc:1034] running on GCE node
I20260810 23:26:00.165118 25489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:26:00.166837 25489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:26:00.168089 25489 hybrid_clock.cc:648] HybridClock initialized: now 1786404360168015 us; error 60 us; skew 500 ppm
I20260810 23:26:00.168476 25489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:26:00 dist-test-slave-rmhg krb5kdc[24524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404360, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:26:00.171999 25489 init.cc:377] Logged in from keytab as oryx/127.14.175.124@KRBTEST.COM (short username oryx)
I20260810 23:26:00.173764 25489 webserver.cc:460] Webserver started at http://127.14.175.124:43075/ using document root <none> and password file <none>
I20260810 23:26:00.174505 25489 fs_manager.cc:359] Metadata directory not provided
I20260810 23:26:00.174630 25489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:26:00.174952 25489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:26:00.178185 25489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-2/data/instance:
uuid: "23fb6044cc6444828df1c91f64d20796"
format_stamp: "Formatted at 2026-08-10 23:26:00 on dist-test-slave-rmhg"
server_key: "269855db218a45897b7a64895ec07f1d"
server_key_iv: "5b38d6f5e82102eefdf3f7b6fa654b0d"
server_key_version: "encryptionkey@0"
I20260810 23:26:00.178982 25489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-2/wal/instance:
uuid: "23fb6044cc6444828df1c91f64d20796"
format_stamp: "Formatted at 2026-08-10 23:26:00 on dist-test-slave-rmhg"
server_key: "269855db218a45897b7a64895ec07f1d"
server_key_iv: "5b38d6f5e82102eefdf3f7b6fa654b0d"
server_key_version: "encryptionkey@0"
I20260810 23:26:00.184598 25489 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:26:00.188377 25659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:00.189893 25489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:26:00.190065 25489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-2/wal
uuid: "23fb6044cc6444828df1c91f64d20796"
format_stamp: "Formatted at 2026-08-10 23:26:00 on dist-test-slave-rmhg"
server_key: "269855db218a45897b7a64895ec07f1d"
server_key_iv: "5b38d6f5e82102eefdf3f7b6fa654b0d"
server_key_version: "encryptionkey@0"
I20260810 23:26:00.190275 25489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:26:00.205780 25489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:26:00.210354 25489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:26:00.210640 25489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:26:00.237545 25489 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.124:42991
I20260810 23:26:00.237775 25743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.124:42991 every 8 connection(s)
I20260810 23:26:00.239085 25489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-2/data/info.pb
I20260810 23:26:00.244158 25744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:26:00.245776 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 25489
I20260810 23:26:00.255636 25744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:26:00 dist-test-slave-rmhg krb5kdc[24524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.175.124@KRBTEST.COM
Aug 10 23:26:00 dist-test-slave-rmhg krb5kdc[24524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404360, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.124@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM
I20260810 23:26:00.276706 25748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:00.248294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36254 (local address 127.14.175.124:42991)
0810 23:26:00.249221 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:00.249234 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:00.249303 (+ 69us) server_negotiation.cc:407] Connection header received
0810 23:26:00.251905 (+ 2602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:00.251933 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:00.252430 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:00.252945 (+ 515us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:00.255428 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.257139 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.259247 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.260712 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.260997 (+ 285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:00.267904 (+ 6907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:00.267922 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:00.267961 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:00.268019 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:00.272110 (+ 4091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:00.272953 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:00.272962 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:00.272970 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:00.273196 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:00.273612 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:00.273617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:00.273619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:00.274242 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:00.274580 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:00.275076 (+ 496us) server_negotiation.cc:299] Negotiation successful
0810 23:26:00.275520 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":405,"threads_started":1}
I20260810 23:26:00.279140 25436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:00.259350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36746 (local address 127.14.175.126:46435)
0810 23:26:00.259717 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:00.259724 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:00.261983 (+ 2259us) server_negotiation.cc:407] Connection header received
0810 23:26:00.262293 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:00.262301 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:00.262389 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:00.262482 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:00.263984 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.264772 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.266698 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.267228 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.267348 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:00.272642 (+ 5294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:00.272648 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:00.272651 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:00.272689 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:00.275824 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:00.276758 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:00.276763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:00.276765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:00.276991 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:00.277454 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:00.277458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:00.277460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:00.277860 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:00.277998 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:00.278515 (+ 517us) server_negotiation.cc:299] Negotiation successful
0810 23:26:00.278773 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260810 23:26:00.279320 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:00.259154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.0.0.1:36746)
0810 23:26:00.261830 (+ 2676us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:00.261879 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:00.262133 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:00.262585 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:00.262592 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:00.263475 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:00.263823 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:00.263839 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.264945 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.264951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:00.266548 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:00.266570 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.267583 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.267594 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:00.267948 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:00.268807 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:00.268861 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:00.272408 (+ 3547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:00.275955 (+ 3547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:00.275967 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:00.275975 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:00.276622 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:00.277113 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:00.277118 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:00.277122 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:00.277336 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:00.278009 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:00.278017 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:00.278211 (+ 194us) client_negotiation.cc:769] Sending connection context
0810 23:26:00.278383 (+ 172us) client_negotiation.cc:240] Negotiation successful
0810 23:26:00.278755 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1957,"spinlock_wait_cycles":25344,"thread_start_us":1399,"threads_started":1}
I20260810 23:26:00.280722 25744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:26:00 dist-test-slave-rmhg krb5kdc[24524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404360, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.124@KRBTEST.COM for oryx/127.14.175.125@KRBTEST.COM
I20260810 23:26:00.297577 25750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:00.281529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42218 (local address 127.14.175.125:32771)
0810 23:26:00.282304 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:00.282309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:00.282326 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:26:00.282375 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:00.282379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:00.282438 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:00.282529 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:00.283456 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.284584 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.286290 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.287028 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.287175 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:00.291172 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:00.291178 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:00.291182 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:00.291222 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:00.294923 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:00.295744 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:00.295749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:00.295751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:00.295904 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:00.296253 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:00.296257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:00.296259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:00.296627 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:00.296785 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:00.297074 (+ 289us) server_negotiation.cc:299] Negotiation successful
0810 23:26:00.297280 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":462,"threads_started":1}
I20260810 23:26:00.297623 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:00.281411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:32771 (local address 127.0.0.1:42218)
0810 23:26:00.281601 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:00.281620 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:00.281723 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:00.282563 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:00.282567 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:00.283057 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:00.283314 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:00.283324 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.284739 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.284743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:00.286142 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:00.286157 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.287194 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.287199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:00.287505 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:00.288093 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:00.288125 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:00.290892 (+ 2767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:00.295093 (+ 4201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:00.295098 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:00.295101 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:00.295625 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:00.295990 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:00.295994 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:00.295996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:00.296157 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:00.296716 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:00.296719 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:00.296872 (+ 153us) client_negotiation.cc:769] Sending connection context
0810 23:26:00.296979 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 23:26:00.297298 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 23:26:00.299103 25744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42991 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:26:00.304397 25751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: NON_VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: true } } }
I20260810 23:26:00.305959 25751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [sys.catalog]: This master's current role is: LEADER
Aug 10 23:26:00 dist-test-slave-rmhg krb5kdc[24524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404360, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.124@KRBTEST.COM for oryx/127.14.175.124@KRBTEST.COM
I20260810 23:26:00.316848 25750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:00.305345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42220 (local address 127.14.175.125:32771)
0810 23:26:00.305608 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:00.305614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:00.308497 (+ 2883us) server_negotiation.cc:407] Connection header received
0810 23:26:00.308705 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:00.308711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:00.308861 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:00.308994 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:26:00.309634 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.310696 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.313587 (+ 2891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.315413 (+ 1826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.315643 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:00.315969 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:00.316342 (+ 373us) server_negotiation.cc:299] Negotiation successful
0810 23:26:00.316526 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 23:26:00.316860 25754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:00.304556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:32771 (local address 127.0.0.1:42220)
0810 23:26:00.308364 (+ 3808us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:00.308385 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:00.308544 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:00.309050 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:00.309055 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:00.309083 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:26:00.309440 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:00.309454 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.310880 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.310884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:00.313339 (+ 2455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:00.313361 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.315674 (+ 2313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.315682 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:00.316080 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:00.316172 (+ 92us) client_negotiation.cc:769] Sending connection context
0810 23:26:00.316279 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 23:26:00.316447 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3631,"thread_start_us":3304,"threads_started":1}
I20260810 23:26:00.318478 25615 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79ba9107e6d44ecf90ec45f01d3ec450. Latest consensus state: current_term: 1 leader_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: NON_VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: true } } }
I20260810 23:26:00.318847 25615 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [sys.catalog]: This master's current role is: LEARNER
I20260810 23:26:00.319648 25748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:00.299877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36256 (local address 127.14.175.124:42991)
0810 23:26:00.300021 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:00.300026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:00.300043 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:26:00.300190 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:00.300195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:00.300246 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:00.300318 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:00.301102 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.302229 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.303996 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.304776 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.304928 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:00.311711 (+ 6783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:00.311718 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:00.311723 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:00.311765 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:00.315767 (+ 4002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:00.316727 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:00.316733 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:00.316737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:00.316898 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:00.317564 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:00.317569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:00.317572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:00.318035 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:00.318186 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:00.318931 (+ 745us) server_negotiation.cc:299] Negotiation successful
0810 23:26:00.319214 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 23:26:00.320109 25752 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450: attempting to promote NON_VOTER c8e7587e7a104723916c49843b278e0f to VOTER
I20260810 23:26:00.320672 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:00.299796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:42991 (local address 127.0.0.1:36256)
0810 23:26:00.299968 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:00.299988 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:00.300086 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:00.300347 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:00.300351 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:00.300784 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:00.300973 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:00.300982 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.302381 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.302385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:00.303853 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:00.303868 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.306757 (+ 2889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.306763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:00.307171 (+ 408us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:00.308123 (+ 952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:00.308141 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:00.311469 (+ 3328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:00.315982 (+ 4513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:00.315987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:00.315992 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:00.316596 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:00.317172 (+ 576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:00.317177 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:00.317180 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:00.317409 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:00.318224 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:00.318229 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:00.318465 (+ 236us) client_negotiation.cc:769] Sending connection context
0810 23:26:00.319949 (+ 1484us) client_negotiation.cc:240] Negotiation successful
0810 23:26:00.320236 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 23:26:00.321424 25752 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } }
I20260810 23:26:00.324015 25561 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 1 LEARNER]: Refusing update from remote peer 79ba9107e6d44ecf90ec45f01d3ec450: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 23:26:00.324704 25752 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 23:26:00.327812 25744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796: Bootstrap starting.
I20260810 23:26:00.328718 25751 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8e7587e7a104723916c49843b278e0f (127.14.175.125) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } }
I20260810 23:26:00.332449 25744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796: Neither blocks nor log segments found. Creating new log.
I20260810 23:26:00.333362 25753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } }
I20260810 23:26:00.334054 25753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [sys.catalog]: This master's current role is: LEADER
I20260810 23:26:00.333377 25561 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8e7587e7a104723916c49843b278e0f (127.14.175.125) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } }
I20260810 23:26:00.334380 25744 log.cc:824] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796: Log is configured to *not* fsync() on all Append() calls
I20260810 23:26:00.334568 25753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } }
I20260810 23:26:00.334879 25753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [sys.catalog]: This master's current role is: LEADER
I20260810 23:26:00.337555 25615 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } }
I20260810 23:26:00.337810 25615 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:26:00.338966 25744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796: No bootstrap required, opened a new log
I20260810 23:26:00.350220 25744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42991 } }
I20260810 23:26:00.351025 25744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:26:00.351277 25744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23fb6044cc6444828df1c91f64d20796, State: Initialized, Role: FOLLOWER
I20260810 23:26:00.352038 25744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42991 } }
I20260810 23:26:00.353652 25762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42991 } } }
I20260810 23:26:00.354352 25744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:26:00.354389 25762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 23:26:00.370810 25786 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:26:00.371024 25786 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:26:00.374380 25489 master_runner.cc:428] Detected that this master 23fb6044cc6444828df1c91f64d20796 is joining an existing cluster
I20260810 23:26:00.374627 25489 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.175.124:42991
I20260810 23:26:00.386598 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 24687
Aug 10 23:26:00 dist-test-slave-rmhg krb5kdc[24524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.175.65@KRBTEST.COM
I20260810 23:26:00.426553 25798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:00.395080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56916 (local address 127.14.175.65:35079)
0810 23:26:00.395902 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:00.395912 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:00.395946 (+ 34us) server_negotiation.cc:407] Connection header received
0810 23:26:00.396020 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:00.396029 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:00.396357 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:00.398650 (+ 2293us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:00.399070 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.400890 (+ 1820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.403960 (+ 3070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.407760 (+ 3800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.408901 (+ 1141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:00.416673 (+ 7772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:00.416702 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:00.416728 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:00.416778 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:00.421720 (+ 4942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:00.422772 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:00.422784 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:00.422787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:00.422979 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:00.423521 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:00.423526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:00.423530 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:00.424029 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:00.424349 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:00.425662 (+ 1313us) server_negotiation.cc:299] Negotiation successful
0810 23:26:00.425960 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":837,"threads_started":1}
I20260810 23:26:00.427409 25436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:00.393623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36752 (local address 127.14.175.126:46435)
0810 23:26:00.394805 (+ 1182us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:00.394812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:00.395450 (+ 638us) server_negotiation.cc:407] Connection header received
0810 23:26:00.395632 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:00.395639 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:00.395712 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:00.395794 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:00.402039 (+ 6245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.403691 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.410564 (+ 6873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.411776 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.411905 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:00.418519 (+ 6614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:00.418528 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:00.418533 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:00.418583 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:00.423203 (+ 4620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:00.424088 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:00.424094 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:00.424098 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:00.424280 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:00.424971 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:00.424976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:00.424979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:00.425648 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:00.425787 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:00.426722 (+ 935us) server_negotiation.cc:299] Negotiation successful
0810 23:26:00.426977 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1028}
I20260810 23:26:00.428658 24883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:26:00.428972 25797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:00.393696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.0.0.1:36752)
0810 23:26:00.395273 (+ 1577us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:00.395294 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:00.395494 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:00.395854 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:00.395858 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:00.400829 (+ 4971us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:00.401758 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:00.401783 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.404114 (+ 2331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.404121 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:00.410268 (+ 6147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:00.410300 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.412465 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.412481 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:00.412849 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:00.414045 (+ 1196us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:00.414071 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:00.418140 (+ 4069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:00.423381 (+ 5241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:00.423385 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:00.423388 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:00.423925 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:00.424510 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:00.424514 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:00.424517 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:00.424833 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:00.425917 (+ 1084us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:00.425922 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:00.426216 (+ 294us) client_negotiation.cc:769] Sending connection context
0810 23:26:00.427658 (+ 1442us) client_negotiation.cc:240] Negotiation successful
0810 23:26:00.428127 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1342,"thread_start_us":948,"threads_started":1}
I20260810 23:26:00.442456 25750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:00.394551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42228 (local address 127.14.175.125:32771)
0810 23:26:00.395401 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:00.395407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:00.416688 (+ 21281us) server_negotiation.cc:407] Connection header received
0810 23:26:00.416818 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:00.416824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:00.416912 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:00.417002 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:00.419510 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.420743 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.423672 (+ 2929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.424374 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.424515 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:00.435908 (+ 11393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:00.435917 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:00.435921 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:00.435968 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:00.439272 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:00.439947 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:00.439951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:00.439953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:00.440114 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:00.440509 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:00.440513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:00.440515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:00.440822 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:00.440980 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:00.441636 (+ 656us) server_negotiation.cc:299] Negotiation successful
0810 23:26:00.441951 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717}
I20260810 23:26:00.443809 25799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:00.397082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:32771 (local address 127.0.0.1:42228)
0810 23:26:00.416564 (+ 19482us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:00.416582 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:00.416703 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:00.417320 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:00.417325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:00.418623 (+ 1298us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:00.419355 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:00.419368 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.421211 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.421217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:00.423499 (+ 2282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:00.423514 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.433256 (+ 9742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.433268 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:00.433478 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:00.434084 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:00.434108 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:00.435677 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:00.439402 (+ 3725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:00.439407 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:00.439412 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:00.439815 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:00.440225 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:00.440229 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:00.440231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:00.440395 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:00.440943 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:00.440948 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:00.441230 (+ 282us) client_negotiation.cc:769] Sending connection context
0810 23:26:00.442692 (+ 1462us) client_negotiation.cc:240] Negotiation successful
0810 23:26:00.443110 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19082,"thread_start_us":15247,"threads_started":1}
I20260810 23:26:00.457691 25762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:26:00.458235 25762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42991 } }
I20260810 23:26:00.459571 25801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:00.396677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:42991 (local address 127.0.0.1:36260)
0810 23:26:00.413587 (+ 16910us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:00.413611 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:00.414840 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:00.415444 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:00.415449 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:00.425416 (+ 9967us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:00.426122 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:00.426135 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.427824 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.427829 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:00.433894 (+ 6065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:00.433914 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.435515 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.435522 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:00.435713 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:00.445076 (+ 9363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:00.445099 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:00.446470 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:00.452594 (+ 6124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:00.452600 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:00.452605 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:00.453430 (+ 825us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:00.453962 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:00.453967 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:00.453971 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:00.454184 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:00.455658 (+ 1474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:00.455662 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:00.455898 (+ 236us) client_negotiation.cc:769] Sending connection context
0810 23:26:00.456041 (+ 143us) client_negotiation.cc:240] Negotiation successful
0810 23:26:00.456436 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15393,"thread_start_us":19071,"threads_started":1}
I20260810 23:26:00.459595 25748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:00.396950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36260 (local address 127.14.175.124:42991)
0810 23:26:00.397184 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:00.397189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:00.413687 (+ 16498us) server_negotiation.cc:407] Connection header received
0810 23:26:00.415086 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:00.415092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:00.415208 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:00.415405 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:00.426284 (+ 10879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.427557 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.434083 (+ 6526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.434935 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.435061 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:00.446714 (+ 11653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:00.446721 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:00.446726 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:00.446766 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:00.452354 (+ 5588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:00.453608 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:00.453614 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:00.453618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:00.453828 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:00.454315 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:00.454320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:00.454323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:00.454908 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:00.455082 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:00.456238 (+ 1156us) server_negotiation.cc:299] Negotiation successful
0810 23:26:00.457408 (+ 1170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":166,"server-negotiator.queue_time_us":61}
I20260810 23:26:00.460750 25762 leader_election.cc:288] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79ba9107e6d44ecf90ec45f01d3ec450 (127.14.175.126:46435), c8e7587e7a104723916c49843b278e0f (127.14.175.125:32771)
I20260810 23:26:00.461426 25561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "23fb6044cc6444828df1c91f64d20796" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8e7587e7a104723916c49843b278e0f" is_pre_election: true
I20260810 23:26:00.462162 24604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "23fb6044cc6444828df1c91f64d20796" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" is_pre_election: true
W20260810 23:26:00.463397 25667 tablet.cc:2369] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:26:00.463371 25660 leader_election.cc:302] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23fb6044cc6444828df1c91f64d20796; no voters: 79ba9107e6d44ecf90ec45f01d3ec450, c8e7587e7a104723916c49843b278e0f
I20260810 23:26:00.466146 25762 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Aug 10 23:26:00 dist-test-slave-rmhg krb5kdc[24524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404357, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.126@KRBTEST.COM for oryx/127.14.175.124@KRBTEST.COM
I20260810 23:26:00.504997 25805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:00.436657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36262 (local address 127.14.175.124:42991)
0810 23:26:00.456274 (+ 19617us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:00.456280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:00.456315 (+ 35us) server_negotiation.cc:407] Connection header received
0810 23:26:00.456439 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:00.456443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:00.456652 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:00.457659 (+ 1007us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:00.460246 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.462641 (+ 2395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.464565 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.465084 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.465223 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:00.469014 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:00.469020 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:00.469024 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:00.469057 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:00.501310 (+ 32253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:00.502451 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:00.502457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:00.502461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:00.502672 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:00.503151 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:00.503157 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:00.503161 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:00.503601 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:00.503753 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:00.504244 (+ 491us) server_negotiation.cc:299] Negotiation successful
0810 23:26:00.504530 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19323,"thread_start_us":18958,"threads_started":1}
I20260810 23:26:00.504976 25754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:00.435846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:42991 (local address 127.0.0.1:36262)
0810 23:26:00.436867 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:00.436886 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:00.436992 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:00.457024 (+ 20032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:00.457031 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:00.457830 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:00.458110 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:00.458126 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.462911 (+ 4785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.462917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:00.464351 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:00.464372 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.465250 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.465256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:00.465461 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:00.466102 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:00.466120 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:00.468742 (+ 2622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:00.501597 (+ 32855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:00.501604 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:00.501610 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:00.502273 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:00.502810 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:00.502816 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:00.502821 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:00.503019 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:00.503740 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:00.503745 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:00.503948 (+ 203us) client_negotiation.cc:769] Sending connection context
0810 23:26:00.504103 (+ 155us) client_negotiation.cc:240] Negotiation successful
0810 23:26:00.504455 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":888}
I20260810 23:26:00.507011 24590 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.14.175.124:42991: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "23fb6044cc6444828df1c91f64d20796"
member_type: NON_VOTER
last_known_addr {
host: "127.14.175.124"
port: 42991
}
attrs {
promote: true
}
}
dest_uuid: "79ba9107e6d44ecf90ec45f01d3ec450"
cas_config_opid_index: 6
I20260810 23:26:00.507861 24590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: NON_VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: true } }
I20260810 23:26:00.510679 25702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:26:00.513643 25561 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 1 FOLLOWER]: Refusing update from remote peer 79ba9107e6d44ecf90ec45f01d3ec450: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 23:26:00.514508 25753 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 23:26:00.515743 25702 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [term 1 FOLLOWER]: Refusing update from remote peer 79ba9107e6d44ecf90ec45f01d3ec450: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 23:26:00.516548 25753 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: NON_VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:26:00.518139 25751 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 23fb6044cc6444828df1c91f64d20796 (127.14.175.124) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: NON_VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: true } } }
I20260810 23:26:00.522053 25561 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 23fb6044cc6444828df1c91f64d20796 (127.14.175.124) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: NON_VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: true } } }
I20260810 23:26:00.527485 25751 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.14.175.124:42991
I20260810 23:26:00.528374 25489 master.cc:537] Master@127.14.175.124:42991 shutting down...
I20260810 23:26:00.530009 25615 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: NON_VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: true } } }
I20260810 23:26:00.530436 25615 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:26:00.530691 25751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: NON_VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: true } } }
I20260810 23:26:00.530917 25752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: NON_VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: true } } }
I20260810 23:26:00.531176 25751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [sys.catalog]: This master's current role is: LEADER
I20260810 23:26:00.531325 25752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [sys.catalog]: This master's current role is: LEADER
I20260810 23:26:00.536178 25702 raft_consensus.cc:858] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42991 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: NON_VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } }
W20260810 23:26:00.536386 25702 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42991 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: NON_VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } }
I20260810 23:26:00.549211 25702 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c8e7587e7a104723916c49843b278e0f (127.14.175.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: NON_VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: true } } }
I20260810 23:26:00.554114 25814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: NON_VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: true } } }
I20260810 23:26:00.554677 25762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79ba9107e6d44ecf90ec45f01d3ec450. Latest consensus state: current_term: 1 leader_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: NON_VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: true } } }
I20260810 23:26:00.554939 25762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260810 23:26:00.554668 25814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260810 23:26:00.554936 25810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: NON_VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: true } } }
I20260810 23:26:00.555385 25810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260810 23:26:00.557430 25751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: NON_VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: true } } }
I20260810 23:26:00.557750 25751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [sys.catalog]: This master's current role is: LEADER
W20260810 23:26:00.558717 24564 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260810 23:26:00.559584 25489 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260810 23:26:00.560128 25489 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260810 23:26:00.560324 25489 tablet_replica.cc:330] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796: stopping tablet replica
W20260810 23:26:00.566215 24564 consensus_peers.cc:487] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 -> Peer 23fb6044cc6444828df1c91f64d20796 (127.14.175.124:42991): Couldn't send request to peer 23fb6044cc6444828df1c91f64d20796. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:26:00.574792 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 24687
I20260810 23:26:00.579609 25489 master.cc:552] Master@127.14.175.124:42991 shutdown complete.
I20260810 23:26:00.579720 25489 master_runner.cc:305] Clearing existing system tablet
I20260810 23:26:00.580183 25489 fs_manager.cc:359] Metadata directory not provided
I20260810 23:26:00.580313 25489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:26:00.584923 25489 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:26:00.586705 25819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:00.587098 25489 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:26:00.587354 25489 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:26:00.587461 25489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-2/wal
uuid: "23fb6044cc6444828df1c91f64d20796"
format_stamp: "Formatted at 2026-08-10 23:26:00 on dist-test-slave-rmhg"
server_key: "269855db218a45897b7a64895ec07f1d"
server_key_iv: "5b38d6f5e82102eefdf3f7b6fa654b0d"
server_key_version: "encryptionkey@0"
I20260810 23:26:00.591161 25489 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:26:00.596225 25489 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 23:26:00.596410 25489 log.cc:1196] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 23:26:00.596887 25489 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796: Deleting consensus metadata
I20260810 23:26:00.597128 25489 master_runner.cc:315] Copying system tablet from 127.14.175.126:46435
I20260810 23:26:00.599905 25489 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796: tablet copy: Beginning tablet copy session from remote peer at address 127.14.175.126:46435
I20260810 23:26:00.610759 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.65:35079
--local_ip_for_outbound_sockets=127.14.175.65
--tserver_master_addrs=127.14.175.126:46435
--webserver_port=42401
--webserver_interface=127.14.175.65
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.65.keytab
--principal=oryx/127.14.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:36377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.175.126:46435,127.14.175.125:32771
--tserver_master_addrs=127.14.175.126:46435,127.14.175.125:32771,127.14.175.124:42991 with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:26:00.613587 25436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:00.600758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36756 (local address 127.14.175.126:46435)
0810 23:26:00.601027 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:00.601033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:00.601576 (+ 543us) server_negotiation.cc:407] Connection header received
0810 23:26:00.601761 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:00.601767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:00.601856 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:00.601945 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:00.602824 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.603585 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.604880 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.605394 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.605512 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:00.607396 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:00.607402 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:00.607405 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:00.607440 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:00.610330 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:00.611285 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:00.611291 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:00.611295 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:00.611459 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:00.611966 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:00.611970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:00.611973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:00.612352 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:00.612519 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:00.612982 (+ 463us) server_negotiation.cc:299] Negotiation successful
0810 23:26:00.613267 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260810 23:26:00.613745 25831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:00.600684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.0.0.1:36756)
0810 23:26:00.601437 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:00.601467 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:00.601631 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:00.602002 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:00.602007 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:00.602447 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:00.602666 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:00.602676 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.603743 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.603747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:00.604733 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:00.604746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:00.605557 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:00.605563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:00.605740 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:00.606394 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:00.606412 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:00.607161 (+ 749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:00.610515 (+ 3354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:00.610522 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:00.610527 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:00.611104 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:00.611595 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:00.611600 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:00.611604 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:00.611842 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:00.612484 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:00.612490 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:00.612704 (+ 214us) client_negotiation.cc:769] Sending connection context
0810 23:26:00.612868 (+ 164us) client_negotiation.cc:240] Negotiation successful
0810 23:26:00.613309 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":506,"threads_started":1}
I20260810 23:26:00.614533 24620 tablet_copy_service.cc:140] P 79ba9107e6d44ecf90ec45f01d3ec450: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 23fb6044cc6444828df1c91f64d20796 ({username='oryx', principal='oryx/127.14.175.124@KRBTEST.COM'} at 127.0.0.1:36756)
I20260810 23:26:00.614738 24620 tablet_copy_service.cc:161] P 79ba9107e6d44ecf90ec45f01d3ec450: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 23fb6044cc6444828df1c91f64d20796 at {username='oryx', principal='oryx/127.14.175.124@KRBTEST.COM'} at 127.0.0.1:36756: session id = 23fb6044cc6444828df1c91f64d20796-00000000000000000000000000000000
I20260810 23:26:00.618461 24620 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 23:26:00.620690 25489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 23:26:00.627944 25489 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796: tablet copy: Starting download of 0 data blocks...
I20260810 23:26:00.628289 25489 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796: tablet copy: Starting download of 1 WAL segments...
I20260810 23:26:00.631348 25489 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 23:26:00.636499 24620 tablet_copy_service.cc:342] P 79ba9107e6d44ecf90ec45f01d3ec450: Request end of tablet copy session 23fb6044cc6444828df1c91f64d20796-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.175.124@KRBTEST.COM'} at 127.0.0.1:36756
I20260810 23:26:00.636694 24620 tablet_copy_service.cc:433] P 79ba9107e6d44ecf90ec45f01d3ec450: ending tablet copy session 23fb6044cc6444828df1c91f64d20796-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 23fb6044cc6444828df1c91f64d20796
I20260810 23:26:00.650799 25489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:26:00.653853 25834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:00.658712 25835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:26:00.665685 25489 server_base.cc:1034] running on GCE node
W20260810 23:26:00.666033 25838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:26:00.666663 25489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:26:00.667446 25489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:26:00.668670 25489 hybrid_clock.cc:648] HybridClock initialized: now 1786404360668612 us; error 52 us; skew 500 ppm
I20260810 23:26:00.668970 25489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:26:00 dist-test-slave-rmhg krb5kdc[24524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404360, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:26:00.671226 25489 init.cc:377] Logged in from keytab as oryx/127.14.175.124@KRBTEST.COM (short username oryx)
I20260810 23:26:00.674202 25489 webserver.cc:460] Webserver started at http://127.14.175.124:43687/ using document root <none> and password file <none>
I20260810 23:26:00.674639 25489 fs_manager.cc:359] Metadata directory not provided
I20260810 23:26:00.674750 25489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:26:00.678042 25489 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:26:00.680110 25850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:00.680584 25489 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260810 23:26:00.680716 25489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-2/wal
uuid: "23fb6044cc6444828df1c91f64d20796"
format_stamp: "Formatted at 2026-08-10 23:26:00 on dist-test-slave-rmhg"
server_key: "269855db218a45897b7a64895ec07f1d"
server_key_iv: "5b38d6f5e82102eefdf3f7b6fa654b0d"
server_key_version: "encryptionkey@0"
I20260810 23:26:00.680905 25489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:26:00.706674 25489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:26:00.711277 25489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:26:00.747934 25489 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.124:42991
I20260810 23:26:00.747995 25931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.124:42991 every 8 connection(s)
I20260810 23:26:00.749022 25489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/master-2/data/info.pb
I20260810 23:26:00.754899 25932 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:26:00.759302 25932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796: Bootstrap starting.
I20260810 23:26:00.778185 25932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:26:00.778900 25932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796: Bootstrap complete.
I20260810 23:26:00.780153 25932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: NON_VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: true } }
I20260810 23:26:00.780515 25932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 23fb6044cc6444828df1c91f64d20796, State: Initialized, Role: LEARNER
I20260810 23:26:00.780844 25932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: NON_VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: true } }
I20260810 23:26:00.781920 25937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: NON_VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: true } } }
I20260810 23:26:00.782503 25937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [sys.catalog]: This master's current role is: LEARNER
I20260810 23:26:00.782351 25932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:26:00.798193 25957 catalog_manager.cc:1259] Loaded cluster ID: 8d5f154f21774aaa952ac2fb0c6079bd
I20260810 23:26:00.798333 25957 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796: loading cluster ID for follower catalog manager: success
I20260810 23:26:00.802266 25957 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796: acquiring CA information for follower catalog manager: success
I20260810 23:26:00.805116 25957 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 23:26:01.052898 25832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:26:01.053294 25832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:26:01.053476 25832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:26:01.062639 25832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:26:01.062810 25832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:26:01.062966 25832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.65
I20260810 23:26:01.075758 25832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:36377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.65.keytab
--principal=oryx/127.14.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.65:35079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.175.65
--webserver_port=42401
--tserver_master_addrs=127.14.175.126:46435,127.14.175.125:32771,127.14.175.124:42991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.65
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:26:01.077436 25832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:26:01.078996 25832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:26:01.097638 25966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:01.097846 25969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:01.099576 25965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:26:01.099704 25832 server_base.cc:1034] running on GCE node
I20260810 23:26:01.100443 25832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:26:01.101598 25832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:26:01.102819 25832 hybrid_clock.cc:648] HybridClock initialized: now 1786404361102738 us; error 59 us; skew 500 ppm
I20260810 23:26:01.103251 25832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:26:01 dist-test-slave-rmhg krb5kdc[24524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404361, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:26:01.108338 25832 init.cc:377] Logged in from keytab as oryx/127.14.175.65@KRBTEST.COM (short username oryx)
I20260810 23:26:01.110389 25832 webserver.cc:460] Webserver started at http://127.14.175.65:42401/ using document root <none> and password file <none>
I20260810 23:26:01.111209 25832 fs_manager.cc:359] Metadata directory not provided
I20260810 23:26:01.111380 25832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:26:01.118530 25832 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 23:26:01.122876 25979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:01.124770 25832 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260810 23:26:01.124981 25832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-0/wal
uuid: "4fd367b2b2444451b116d2d69f690b77"
format_stamp: "Formatted at 2026-08-10 23:25:57 on dist-test-slave-rmhg"
server_key: "26c18b4df06249abb0df4d7bded8a3a5"
server_key_iv: "dec8f7ca742a1df59e3cb6fe9a1daf04"
server_key_version: "encryptionkey@0"
I20260810 23:26:01.126716 25832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:26:01.154361 25832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:26:01.160130 25832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:26:01.160554 25832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:26:01.162061 25832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:26:01.164070 25832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:26:01.164252 25832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:01.164395 25832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:26:01.164553 25832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:01.170989 25751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: NON_VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: true } } }
I20260810 23:26:01.171388 25751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [sys.catalog]: This master's current role is: LEADER
I20260810 23:26:01.192327 25996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:01.172998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36272 (local address 127.14.175.124:42991)
0810 23:26:01.182438 (+ 9440us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:01.182444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:01.182466 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:26:01.182559 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:01.182564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:01.182805 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:01.182928 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:26:01.183546 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:01.184922 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:01.187988 (+ 3066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:01.189811 (+ 1823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:01.190118 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:01.190390 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:01.191664 (+ 1274us) server_negotiation.cc:299] Negotiation successful
0810 23:26:01.191844 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9315,"thread_start_us":9165,"threads_started":1}
I20260810 23:26:01.192775 25995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:01.172577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:42991 (local address 127.0.0.1:36272)
0810 23:26:01.181483 (+ 8906us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:01.181511 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:01.181714 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:01.183019 (+ 1305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:01.183026 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:01.183069 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:26:01.183357 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:01.183373 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:01.185213 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:01.185220 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:01.187670 (+ 2450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:01.187695 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:01.190657 (+ 2962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:01.190665 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:01.190921 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:01.190966 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 23:26:01.191088 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 23:26:01.191276 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8678,"thread_start_us":8500,"threads_started":1}
I20260810 23:26:01.194108 25893 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260810 23:26:01.194552 25937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79ba9107e6d44ecf90ec45f01d3ec450. Latest consensus state: current_term: 1 leader_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: NON_VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: true } } }
I20260810 23:26:01.194841 25937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [sys.catalog]: This master's current role is: LEARNER
I20260810 23:26:01.195523 25751 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450: attempting to promote NON_VOTER 23fb6044cc6444828df1c91f64d20796 to VOTER
I20260810 23:26:01.196683 25751 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: false } }
I20260810 23:26:01.199199 25561 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 1 FOLLOWER]: Refusing update from remote peer 79ba9107e6d44ecf90ec45f01d3ec450: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260810 23:26:01.199807 25893 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [term 1 LEARNER]: Refusing update from remote peer 79ba9107e6d44ecf90ec45f01d3ec450: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260810 23:26:01.200156 25988 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260810 23:26:01.200698 25990 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260810 23:26:01.203828 25988 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 23fb6044cc6444828df1c91f64d20796 (127.14.175.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: false } } }
I20260810 23:26:01.204567 25561 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 23fb6044cc6444828df1c91f64d20796 (127.14.175.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: false } } }
I20260810 23:26:01.209846 25751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: false } } }
I20260810 23:26:01.210211 25751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [sys.catalog]: This master's current role is: LEADER
I20260810 23:26:01.210551 25751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: false } } }
I20260810 23:26:01.210855 25751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [sys.catalog]: This master's current role is: LEADER
I20260810 23:26:01.216681 25751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: false } } }
I20260810 23:26:01.217051 25751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450 [sys.catalog]: This master's current role is: LEADER
I20260810 23:26:01.217904 25892 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 23fb6044cc6444828df1c91f64d20796 (127.14.175.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: false } } }
I20260810 23:26:01.218796 26052 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: false } } }
I20260810 23:26:01.219426 26052 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:26:01.223809 25937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: false } } }
I20260810 23:26:01.224040 25937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:26:01.267899 25832 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.65:35079
I20260810 23:26:01.268006 26170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.65:35079 every 8 connection(s)
I20260810 23:26:01.269711 25832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
I20260810 23:26:01.276410 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 25832
I20260810 23:26:01.276846 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 24900
Aug 10 23:26:01 dist-test-slave-rmhg krb5kdc[24524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404361, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.124@KRBTEST.COM
Aug 10 23:26:01 dist-test-slave-rmhg krb5kdc[24524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.175.66@KRBTEST.COM
Aug 10 23:26:01 dist-test-slave-rmhg krb5kdc[24524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404361, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.125@KRBTEST.COM
I20260810 23:26:01.314074 25996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:01.273706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:47527 (local address 127.14.175.124:42991)
0810 23:26:01.274260 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:01.274267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:01.282904 (+ 8637us) server_negotiation.cc:407] Connection header received
0810 23:26:01.285689 (+ 2785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:01.285694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:01.285781 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:01.285873 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:01.290737 (+ 4864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:01.292183 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:01.295435 (+ 3252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:01.296249 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:01.296363 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:01.303296 (+ 6933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:01.303307 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:01.303312 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:01.303366 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:01.308394 (+ 5028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:01.309722 (+ 1328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:01.309729 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:01.309732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:01.309985 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:01.310754 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:01.310760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:01.310763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:01.311347 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:01.311502 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:01.312829 (+ 1327us) server_negotiation.cc:299] Negotiation successful
0810 23:26:01.313249 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412}
I20260810 23:26:01.316191 26183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:01.278546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32910 (local address 127.14.175.66:36553)
0810 23:26:01.283278 (+ 4732us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:01.283296 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:01.285103 (+ 1807us) server_negotiation.cc:407] Connection header received
0810 23:26:01.285223 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:01.285233 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:01.285586 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:01.285895 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:01.289563 (+ 3668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:01.291547 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:01.294107 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:01.295345 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:01.295460 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:01.303429 (+ 7969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:01.303457 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:01.303480 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:01.303561 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:01.309064 (+ 5503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:01.310362 (+ 1298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:01.310374 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:01.310378 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:01.312261 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:01.312833 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:01.312838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:01.312841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:01.313431 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:01.313853 (+ 422us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:01.315363 (+ 1510us) server_negotiation.cc:299] Negotiation successful
0810 23:26:01.315740 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4426,"thread_start_us":4204,"threads_started":1}
I20260810 23:26:01.317624 26178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:01.274558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:42991 (local address 127.14.175.65:47527)
0810 23:26:01.282685 (+ 8127us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:01.282747 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:01.285378 (+ 2631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:01.286437 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:01.286450 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:01.287691 (+ 1241us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:01.290414 (+ 2723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:01.290448 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:01.292559 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:01.292565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:01.295241 (+ 2676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:01.295258 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:01.296780 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:01.296790 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:01.297218 (+ 428us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:01.298104 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:01.298141 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:01.302912 (+ 4771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:01.308652 (+ 5740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:01.308663 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:01.308687 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:01.309543 (+ 856us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:01.310173 (+ 630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:01.310178 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:01.310181 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:01.310608 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:01.311686 (+ 1078us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:01.311695 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:01.312303 (+ 608us) client_negotiation.cc:769] Sending connection context
0810 23:26:01.314451 (+ 2148us) client_negotiation.cc:240] Negotiation successful
0810 23:26:01.315238 (+ 787us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7596,"spinlock_wait_cycles":122240,"thread_start_us":19469,"threads_started":1}
I20260810 23:26:01.317768 25083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:26:01.319947 26172 heartbeater.cc:347] Connected to a master server at 127.14.175.124:42991
I20260810 23:26:01.320297 26172 heartbeater.cc:464] Registering TS with master...
I20260810 23:26:01.321698 26172 heartbeater.cc:511] Master 127.14.175.124:42991 requested a full tablet report, sending...
I20260810 23:26:01.325337 26177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:01.273934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:37661 (local address 127.14.175.125:32771)
0810 23:26:01.293712 (+ 19778us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:01.293720 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:01.294919 (+ 1199us) server_negotiation.cc:407] Connection header received
0810 23:26:01.295235 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:01.295241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:01.295327 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:01.295471 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:01.298495 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:01.300242 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:01.303103 (+ 2861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:01.304175 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:01.304288 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:01.312422 (+ 8134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:01.312431 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:01.312436 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:01.312484 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:01.321032 (+ 8548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:01.322319 (+ 1287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:01.322326 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:01.322330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:01.322601 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:01.323230 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:01.323236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:01.323239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:01.323676 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:01.323816 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:01.324526 (+ 710us) server_negotiation.cc:299] Negotiation successful
0810 23:26:01.324787 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19526,"thread_start_us":19781,"threads_started":1}
I20260810 23:26:01.325470 25878 ts_manager.cc:194] Registered new tserver with Master: 4fd367b2b2444451b116d2d69f690b77 (127.14.175.65:35079)
Aug 10 23:26:01 dist-test-slave-rmhg krb5kdc[24524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404361, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM
I20260810 23:26:01.326541 26180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:01.274591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:32771 (local address 127.14.175.65:37661)
0810 23:26:01.294760 (+ 20169us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:01.294798 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:01.295071 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:01.295908 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:01.295913 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:01.297355 (+ 1442us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:01.298264 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:01.298285 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:01.300518 (+ 2233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:01.300523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:01.302878 (+ 2355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:01.302899 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:01.305373 (+ 2474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:01.305381 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:01.305633 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:01.306589 (+ 956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:01.306615 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:01.312103 (+ 5488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:01.321270 (+ 9167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:01.321275 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:01.321279 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:01.322141 (+ 862us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:01.322776 (+ 635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:01.322780 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:01.322783 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:01.323086 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:01.323903 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:01.323908 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:01.324160 (+ 252us) client_negotiation.cc:769] Sending connection context
0810 23:26:01.325619 (+ 1459us) client_negotiation.cc:240] Negotiation successful
0810 23:26:01.325993 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19996,"mutex_wait_us":36,"thread_start_us":7178,"threads_started":1}
I20260810 23:26:01.329437 26174 heartbeater.cc:347] Connected to a master server at 127.14.175.125:32771
I20260810 23:26:01.329617 26174 heartbeater.cc:464] Registering TS with master...
I20260810 23:26:01.330210 26174 heartbeater.cc:511] Master 127.14.175.125:32771 requested a full tablet report, sending...
I20260810 23:26:01.332121 25546 ts_manager.cc:194] Registered new tserver with Master: 4fd367b2b2444451b116d2d69f690b77 (127.14.175.65:35079)
I20260810 23:26:01.337181 26185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:01.301176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.14.175.65:45013)
0810 23:26:01.312335 (+ 11159us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:01.312369 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:01.312592 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:01.313391 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:01.313396 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:01.314734 (+ 1338us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:01.315287 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:01.315301 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:01.317207 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:01.317213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:01.320455 (+ 3242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:01.320476 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:01.322356 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:01.322365 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:01.322705 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:01.323546 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:01.323572 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:01.327781 (+ 4209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:01.333706 (+ 5925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:01.333712 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:01.333716 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:01.334340 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:01.334859 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:01.334866 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:01.334871 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:01.335188 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:01.335884 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:01.335889 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:01.336142 (+ 253us) client_negotiation.cc:769] Sending connection context
0810 23:26:01.336262 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 23:26:01.336563 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10990,"thread_start_us":15741,"threads_started":1}
I20260810 23:26:01.337534 26184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:01.295049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:45013 (local address 127.14.175.126:46435)
0810 23:26:01.311970 (+ 16921us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:01.311976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:01.312717 (+ 741us) server_negotiation.cc:407] Connection header received
0810 23:26:01.312823 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:01.312828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:01.312917 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:01.313023 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:01.315510 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:01.316832 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:01.320719 (+ 3887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:01.321803 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:01.321959 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:01.328226 (+ 6267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:01.328237 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:01.328243 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:01.328299 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:01.333555 (+ 5256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:01.334517 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:01.334524 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:01.334528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:01.334741 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:01.335322 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:01.335327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:01.335330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:01.335755 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:01.335953 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:01.336442 (+ 489us) server_negotiation.cc:299] Negotiation successful
0810 23:26:01.336719 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16716,"thread_start_us":16313,"threads_started":1}
I20260810 23:26:01.338528 26173 heartbeater.cc:347] Connected to a master server at 127.14.175.126:46435
I20260810 23:26:01.338709 26173 heartbeater.cc:464] Registering TS with master...
I20260810 23:26:01.339494 26173 heartbeater.cc:511] Master 127.14.175.126:46435 requested a full tablet report, sending...
I20260810 23:26:01.340947 24590 ts_manager.cc:194] Re-registered known tserver with Master: 4fd367b2b2444451b116d2d69f690b77 (127.14.175.65:35079)
I20260810 23:26:01.341953 24590 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.175.65@KRBTEST.COM'} at 127.14.175.65:45013
I20260810 23:26:01.429981 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 24900
I20260810 23:26:01.451685 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.66:36553
--local_ip_for_outbound_sockets=127.14.175.66
--tserver_master_addrs=127.14.175.126:46435
--webserver_port=35477
--webserver_interface=127.14.175.66
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.66.keytab
--principal=oryx/127.14.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:36377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.175.126:46435,127.14.175.125:32771
--tserver_master_addrs=127.14.175.126:46435,127.14.175.125:32771,127.14.175.124:42991 with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:26:01.811473 26188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:26:01.811820 26188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:26:01.811964 26188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:26:01.820009 26188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:26:01.820156 26188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:26:01.820307 26188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.66
I20260810 23:26:01.830847 26188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:36377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.66.keytab
--principal=oryx/127.14.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.66:36553
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.175.66
--webserver_port=35477
--tserver_master_addrs=127.14.175.126:46435,127.14.175.125:32771,127.14.175.124:42991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.66
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:26:01.832369 26188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:26:01.833926 26188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:26:01.848099 26196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:01.848151 26198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:01.848111 26194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:26:01.849035 26188 server_base.cc:1034] running on GCE node
I20260810 23:26:01.849756 26188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:26:01.850926 26188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:26:01.852173 26188 hybrid_clock.cc:648] HybridClock initialized: now 1786404361852111 us; error 54 us; skew 500 ppm
I20260810 23:26:01.852533 26188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:26:01 dist-test-slave-rmhg krb5kdc[24524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404361, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:26:01.856264 26188 init.cc:377] Logged in from keytab as oryx/127.14.175.66@KRBTEST.COM (short username oryx)
I20260810 23:26:01.858150 26188 webserver.cc:460] Webserver started at http://127.14.175.66:35477/ using document root <none> and password file <none>
I20260810 23:26:01.858899 26188 fs_manager.cc:359] Metadata directory not provided
I20260810 23:26:01.859056 26188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:26:01.865715 26188 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:26:01.869024 26212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:01.870693 26188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:26:01.870893 26188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-1/wal
uuid: "3892f3eb6ebb436d8bcd0f31647a4c6a"
format_stamp: "Formatted at 2026-08-10 23:25:58 on dist-test-slave-rmhg"
server_key: "2b658719537f2ecb4080ba5563ba8718"
server_key_iv: "7948ad53110b7a007d46e9557f5f598b"
server_key_version: "encryptionkey@0"
I20260810 23:26:01.871996 26188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:26:01.885095 26188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:26:01.890308 26188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:26:01.890659 26188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:26:01.891916 26188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:26:01.893524 26188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:26:01.893630 26188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:01.893743 26188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:26:01.893814 26188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:01.929025 26188 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.66:36553
I20260810 23:26:01.929107 26405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.66:36553 every 8 connection(s)
I20260810 23:26:01.930616 26188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
I20260810 23:26:01.938956 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 26188
I20260810 23:26:01.939265 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 25102
Aug 10 23:26:01 dist-test-slave-rmhg krb5kdc[24524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404361, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.124@KRBTEST.COM
Aug 10 23:26:01 dist-test-slave-rmhg krb5kdc[24524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404361, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM
Aug 10 23:26:01 dist-test-slave-rmhg krb5kdc[24524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404361, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.125@KRBTEST.COM
I20260810 23:26:01.963821 26411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:01.933863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:46197 (local address 127.14.175.126:46435)
0810 23:26:01.934313 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:01.934320 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:01.936150 (+ 1830us) server_negotiation.cc:407] Connection header received
0810 23:26:01.938479 (+ 2329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:01.938485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:01.938608 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:01.938719 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:01.941738 (+ 3019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:01.942770 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:01.946099 (+ 3329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:01.946743 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:01.946866 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:01.955013 (+ 8147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:01.955021 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:01.955026 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:01.955073 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:01.959495 (+ 4422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:01.960379 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:01.960386 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:01.960390 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:01.960567 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:01.961284 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:01.961289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:01.961292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:01.961706 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:01.961884 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:01.963117 (+ 1233us) server_negotiation.cc:299] Negotiation successful
0810 23:26:01.963411 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":1023,"threads_started":1}
I20260810 23:26:01.964390 26416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:01.936372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:42991 (local address 127.14.175.66:39781)
0810 23:26:01.936963 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:01.936979 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:01.938161 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:01.940152 (+ 1991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:01.940157 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:01.940774 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:01.941248 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:01.941262 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:01.944520 (+ 3258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:01.944526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:01.946305 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:01.946322 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:01.947622 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:01.947629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:01.947998 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:01.949031 (+ 1033us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:01.949067 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:01.953625 (+ 4558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:01.958560 (+ 4935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:01.958574 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:01.958610 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:01.959427 (+ 817us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:01.960002 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:01.960008 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:01.960012 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:01.960368 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:01.961254 (+ 886us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:01.961266 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:01.961816 (+ 550us) client_negotiation.cc:769] Sending connection context
0810 23:26:01.962175 (+ 359us) client_negotiation.cc:240] Negotiation successful
0810 23:26:01.962736 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":306,"threads_started":1}
I20260810 23:26:01.964682 26417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:01.936529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:39781 (local address 127.14.175.124:42991)
0810 23:26:01.937468 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:01.937472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:01.937490 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:26:01.939720 (+ 2230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:01.939726 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:01.939814 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:01.939921 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:01.943178 (+ 3257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:01.944322 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:01.946506 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:01.947151 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:01.947283 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:01.954487 (+ 7204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:01.954496 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:01.954502 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:01.954554 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:01.958279 (+ 3725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:01.959636 (+ 1357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:01.959643 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:01.959647 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:01.959853 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:01.960531 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:01.960537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:01.960541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:01.960978 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:01.961167 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:01.962832 (+ 1665us) server_negotiation.cc:299] Negotiation successful
0810 23:26:01.964248 (+ 1416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":843,"thread_start_us":739,"threads_started":1}
I20260810 23:26:01.965287 26414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:01.934593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.14.175.66:46197)
0810 23:26:01.935672 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:01.935765 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:01.938152 (+ 2387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:01.939043 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:01.939056 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:01.941310 (+ 2254us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:01.941566 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:01.941586 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:01.942900 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:01.942906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:01.945922 (+ 3016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:01.945942 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:01.947905 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:01.947912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:01.948152 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:01.951078 (+ 2926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:01.951113 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:01.954761 (+ 3648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:01.959691 (+ 4930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:01.959697 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:01.959700 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:01.960233 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:01.960939 (+ 706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:01.960945 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:01.960948 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:01.961122 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:01.962627 (+ 1505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:01.962632 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:01.962852 (+ 220us) client_negotiation.cc:769] Sending connection context
0810 23:26:01.962990 (+ 138us) client_negotiation.cc:240] Negotiation successful
0810 23:26:01.963367 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"mutex_wait_us":1,"spinlock_wait_cycles":13056,"thread_start_us":435,"threads_started":1}
I20260810 23:26:01.966861 26408 heartbeater.cc:347] Connected to a master server at 127.14.175.126:46435
I20260810 23:26:01.966977 26406 heartbeater.cc:347] Connected to a master server at 127.14.175.124:42991
I20260810 23:26:01.967332 26408 heartbeater.cc:464] Registering TS with master...
I20260810 23:26:01.967340 26406 heartbeater.cc:464] Registering TS with master...
I20260810 23:26:01.967726 26412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:01.933887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:46625 (local address 127.14.175.125:32771)
0810 23:26:01.934996 (+ 1109us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:01.935002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:01.935914 (+ 912us) server_negotiation.cc:407] Connection header received
0810 23:26:01.938504 (+ 2590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:01.938510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:01.938606 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:01.938676 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:01.941506 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:01.942657 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:01.949062 (+ 6405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:01.949912 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:01.950036 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:01.955118 (+ 5082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:01.955125 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:01.955130 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:01.955177 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:01.959485 (+ 4308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:01.961909 (+ 2424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:01.961916 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:01.961919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:01.962155 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:01.964255 (+ 2100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:01.964261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:01.964264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:01.965895 (+ 1631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:01.966063 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:01.966736 (+ 673us) server_negotiation.cc:299] Negotiation successful
0810 23:26:01.967111 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":986,"thread_start_us":887,"threads_started":1}
I20260810 23:26:01.968885 26406 heartbeater.cc:511] Master 127.14.175.124:42991 requested a full tablet report, sending...
I20260810 23:26:01.968961 26408 heartbeater.cc:511] Master 127.14.175.126:46435 requested a full tablet report, sending...
I20260810 23:26:01.969296 26415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:01.934595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:32771 (local address 127.14.175.66:46625)
0810 23:26:01.935672 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:01.935765 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:01.938152 (+ 2387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:01.939043 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:01.939056 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:01.940006 (+ 950us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:01.941091 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:01.941123 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:01.942882 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:01.942887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:01.948832 (+ 5945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:01.948848 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:01.950607 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:01.950615 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:01.950827 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:01.951461 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:01.951487 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:01.954410 (+ 2923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:01.960382 (+ 5972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:01.960386 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:01.960390 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:01.961347 (+ 957us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:01.962318 (+ 971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:01.962323 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:01.962327 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:01.962497 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:01.966134 (+ 3637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:01.966139 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:01.966344 (+ 205us) client_negotiation.cc:769] Sending connection context
0810 23:26:01.968179 (+ 1835us) client_negotiation.cc:240] Negotiation successful
0810 23:26:01.968506 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"mutex_wait_us":63,"spinlock_wait_cycles":99840,"thread_start_us":417,"threads_started":1}
I20260810 23:26:01.971015 24590 ts_manager.cc:194] Re-registered known tserver with Master: 3892f3eb6ebb436d8bcd0f31647a4c6a (127.14.175.66:36553)
I20260810 23:26:01.971287 25878 ts_manager.cc:194] Registered new tserver with Master: 3892f3eb6ebb436d8bcd0f31647a4c6a (127.14.175.66:36553)
I20260810 23:26:01.972337 24590 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.175.66@KRBTEST.COM'} at 127.14.175.66:46197
I20260810 23:26:01.972831 26409 heartbeater.cc:347] Connected to a master server at 127.14.175.125:32771
I20260810 23:26:01.973088 26409 heartbeater.cc:464] Registering TS with master...
I20260810 23:26:01.973768 26409 heartbeater.cc:511] Master 127.14.175.125:32771 requested a full tablet report, sending...
I20260810 23:26:01.975427 25546 ts_manager.cc:194] Registered new tserver with Master: 3892f3eb6ebb436d8bcd0f31647a4c6a (127.14.175.66:36553)
Aug 10 23:26:01 dist-test-slave-rmhg krb5kdc[24524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.175.67@KRBTEST.COM
I20260810 23:26:01.989077 26419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:01.940376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38078 (local address 127.14.175.67:44665)
0810 23:26:01.960180 (+ 19804us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:01.960191 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:01.962537 (+ 2346us) server_negotiation.cc:407] Connection header received
0810 23:26:01.962709 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:01.962719 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:01.963016 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:01.963179 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:01.966076 (+ 2897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:01.973027 (+ 6951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:01.974921 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:01.975878 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:01.976032 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:01.981588 (+ 5556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:01.981612 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:01.981633 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:01.981678 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:01.985758 (+ 4080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:01.986693 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:01.986702 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:01.986704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:01.986911 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:01.987350 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:01.987355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:01.987357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:01.987790 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:01.988110 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:01.988364 (+ 254us) server_negotiation.cc:299] Negotiation successful
0810 23:26:01.988637 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19568,"thread_start_us":19828,"threads_started":1}
I20260810 23:26:01.990783 25296 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:26:02.074151 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 25102
W20260810 23:26:02.104056 24566 connection.cc:535] server connection from 127.14.175.67:36357 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 23:26:02.104862 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.67:44665
--local_ip_for_outbound_sockets=127.14.175.67
--tserver_master_addrs=127.14.175.126:46435
--webserver_port=37941
--webserver_interface=127.14.175.67
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.67.keytab
--principal=oryx/127.14.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:36377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.175.126:46435,127.14.175.125:32771
--tserver_master_addrs=127.14.175.126:46435,127.14.175.125:32771,127.14.175.124:42991 with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:26:02.345326 26173 heartbeater.cc:503] Master 127.14.175.126:46435 was elected leader, sending a full tablet report...
W20260810 23:26:02.466986 26421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:26:02.467334 26421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:26:02.467442 26421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:26:02.475042 26421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:26:02.475176 26421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:26:02.475301 26421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.67
I20260810 23:26:02.485826 26421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:36377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.67.keytab
--principal=oryx/127.14.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.67:44665
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.175.67
--webserver_port=37941
--tserver_master_addrs=127.14.175.126:46435,127.14.175.125:32771,127.14.175.124:42991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.67
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:26:02.487207 26421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:26:02.488576 26421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:26:02.502718 26427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:02.502743 26430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:02.502986 26428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:26:02.503577 26421 server_base.cc:1034] running on GCE node
I20260810 23:26:02.504261 26421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:26:02.505334 26421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:26:02.506549 26421 hybrid_clock.cc:648] HybridClock initialized: now 1786404362506494 us; error 46 us; skew 500 ppm
I20260810 23:26:02.506906 26421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:26:02 dist-test-slave-rmhg krb5kdc[24524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404362, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:26:02.510331 26421 init.cc:377] Logged in from keytab as oryx/127.14.175.67@KRBTEST.COM (short username oryx)
I20260810 23:26:02.512034 26421 webserver.cc:460] Webserver started at http://127.14.175.67:37941/ using document root <none> and password file <none>
I20260810 23:26:02.512810 26421 fs_manager.cc:359] Metadata directory not provided
I20260810 23:26:02.512960 26421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:26:02.518901 26421 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260810 23:26:02.522001 26439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:02.523531 26421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:26:02.523715 26421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-2/wal
uuid: "28dabb1e621d4aedbd18cb5ff0091cea"
format_stamp: "Formatted at 2026-08-10 23:25:58 on dist-test-slave-rmhg"
server_key: "d8581afb8589786664aef806ef5cfe52"
server_key_iv: "4d2b2e247c1936519c1602e2cc870be3"
server_key_version: "encryptionkey@0"
I20260810 23:26:02.524838 26421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:26:02.548996 26421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:26:02.553808 26421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:26:02.554106 26421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:26:02.555239 26421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:26:02.556811 26421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:26:02.556911 26421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:02.557015 26421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:26:02.557097 26421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:02.589005 26421 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.67:44665
I20260810 23:26:02.589087 26640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.67:44665 every 8 connection(s)
I20260810 23:26:02.590641 26421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
I20260810 23:26:02.592680 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 26421
Aug 10 23:26:02 dist-test-slave-rmhg krb5kdc[24524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404362, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.124@KRBTEST.COM
I20260810 23:26:02.620370 26652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:02.594445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:49495 (local address 127.14.175.124:42991)
0810 23:26:02.594926 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:02.594934 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:02.596090 (+ 1156us) server_negotiation.cc:407] Connection header received
0810 23:26:02.599009 (+ 2919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:02.599015 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:02.599116 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:02.599197 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:02.601838 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.603004 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.605595 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.606217 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.606316 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:02.611803 (+ 5487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:02.611811 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:02.611816 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:02.611864 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:02.615633 (+ 3769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:02.616902 (+ 1269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:02.616910 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:02.616914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:02.617100 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:02.617763 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:02.617769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:02.617772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:02.618179 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:02.618382 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:02.619650 (+ 1268us) server_negotiation.cc:299] Negotiation successful
0810 23:26:02.619927 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":333,"threads_started":1}
I20260810 23:26:02.623481 26654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:02.594692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:42991 (local address 127.14.175.67:49495)
0810 23:26:02.595889 (+ 1197us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:02.595959 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:02.598645 (+ 2686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:02.599789 (+ 1144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:02.599803 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:02.600683 (+ 880us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:02.601603 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:02.601633 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.603245 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.603250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:02.605390 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:02.605409 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.606741 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.606748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:02.607058 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:02.607976 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:02.608014 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:02.611444 (+ 3430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:02.615918 (+ 4474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:02.615931 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:02.615955 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:02.616714 (+ 759us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:02.617293 (+ 579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:02.617298 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:02.617301 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:02.617622 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:02.618514 (+ 892us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:02.618524 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:02.619081 (+ 557us) client_negotiation.cc:769] Sending connection context
0810 23:26:02.621028 (+ 1947us) client_negotiation.cc:240] Negotiation successful
0810 23:26:02.621820 (+ 792us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"spinlock_wait_cycles":39552,"thread_start_us":2347,"threads_started":1}
I20260810 23:26:02.625970 26641 heartbeater.cc:347] Connected to a master server at 127.14.175.124:42991
I20260810 23:26:02.626365 26641 heartbeater.cc:464] Registering TS with master...
I20260810 23:26:02.627544 26641 heartbeater.cc:511] Master 127.14.175.124:42991 requested a full tablet report, sending...
I20260810 23:26:02.629842 25878 ts_manager.cc:194] Registered new tserver with Master: 28dabb1e621d4aedbd18cb5ff0091cea (127.14.175.67:44665)
Aug 10 23:26:02 dist-test-slave-rmhg krb5kdc[24524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404362, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.125@KRBTEST.COM
Aug 10 23:26:02 dist-test-slave-rmhg krb5kdc[24524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404362, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM
I20260810 23:26:02.655269 26661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:02.595721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:32771 (local address 127.14.175.67:59611)
0810 23:26:02.596324 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:02.596342 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:02.602873 (+ 6531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:02.621363 (+ 18490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:02.621368 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:02.622108 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:02.622401 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:02.622413 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.628742 (+ 6329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.628749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:02.630741 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:02.630762 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.631927 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.631933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:02.632204 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:02.633052 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:02.633072 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:02.636150 (+ 3078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:02.647972 (+ 11822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:02.647978 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:02.647982 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:02.648797 (+ 815us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:02.649424 (+ 627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:02.649429 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:02.649433 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:02.649603 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:02.653956 (+ 4353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:02.653963 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:02.654183 (+ 220us) client_negotiation.cc:769] Sending connection context
0810 23:26:02.654331 (+ 148us) client_negotiation.cc:240] Negotiation successful
0810 23:26:02.654734 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"spinlock_wait_cycles":57088,"thread_start_us":576,"threads_started":1}
I20260810 23:26:02.658321 26662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:02.596179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:59611 (local address 127.14.175.125:32771)
0810 23:26:02.620951 (+ 24772us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:02.620958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:02.620984 (+ 26us) server_negotiation.cc:407] Connection header received
0810 23:26:02.621056 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:02.621062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:02.621187 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:02.621315 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:02.627571 (+ 6256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.628515 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.630963 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.631693 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.631820 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:02.636417 (+ 4597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:02.636423 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:02.636427 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:02.636480 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:02.647821 (+ 11341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:02.648990 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:02.648996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:02.649001 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:02.649270 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:02.649776 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:02.649781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:02.649784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:02.653782 (+ 3998us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:02.657216 (+ 3434us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:02.657414 (+ 198us) server_negotiation.cc:299] Negotiation successful
0810 23:26:02.657683 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24602,"thread_start_us":26268,"threads_started":1}
I20260810 23:26:02.659456 26648 heartbeater.cc:347] Connected to a master server at 127.14.175.125:32771
I20260810 23:26:02.659667 26648 heartbeater.cc:464] Registering TS with master...
I20260810 23:26:02.660511 26648 heartbeater.cc:511] Master 127.14.175.125:32771 requested a full tablet report, sending...
I20260810 23:26:02.661971 26667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:02.597577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.14.175.67:44557)
0810 23:26:02.622003 (+ 24426us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:02.622025 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:02.622149 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:02.639146 (+ 16997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:02.639152 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:02.640473 (+ 1321us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:02.641273 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:02.641297 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.644489 (+ 3192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.644496 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:02.646416 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:02.646440 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.647574 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.647584 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:02.647809 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:02.648579 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:02.648605 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:02.652340 (+ 3735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:02.657589 (+ 5249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:02.657595 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:02.657600 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:02.658217 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:02.658730 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:02.658736 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:02.658740 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:02.658963 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:02.660551 (+ 1588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:02.660557 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:02.660757 (+ 200us) client_negotiation.cc:769] Sending connection context
0810 23:26:02.660918 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 23:26:02.661324 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24272,"thread_start_us":24666,"threads_started":1}
I20260810 23:26:02.661974 26674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:02.597935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:44557 (local address 127.14.175.126:46435)
0810 23:26:02.637775 (+ 39840us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:02.637782 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:02.637805 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:26:02.637873 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:02.637878 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:02.637971 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:02.638090 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:02.642241 (+ 4151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.644206 (+ 1965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.646665 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.647250 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.647407 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:02.652536 (+ 5129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:02.652543 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:02.652547 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:02.652593 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:02.657424 (+ 4831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:02.658400 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:02.658405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:02.658409 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:02.658588 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:02.659105 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:02.659110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:02.659113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:02.659522 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:02.659680 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:02.661239 (+ 1559us) server_negotiation.cc:299] Negotiation successful
0810 23:26:02.661477 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39693,"thread_start_us":40568,"threads_started":1}
I20260810 23:26:02.662685 25546 ts_manager.cc:194] Registered new tserver with Master: 28dabb1e621d4aedbd18cb5ff0091cea (127.14.175.67:44665)
I20260810 23:26:02.664407 26645 heartbeater.cc:347] Connected to a master server at 127.14.175.126:46435
I20260810 23:26:02.664659 26645 heartbeater.cc:464] Registering TS with master...
I20260810 23:26:02.665894 26645 heartbeater.cc:511] Master 127.14.175.126:46435 requested a full tablet report, sending...
I20260810 23:26:02.667311 24590 ts_manager.cc:194] Re-registered known tserver with Master: 28dabb1e621d4aedbd18cb5ff0091cea (127.14.175.67:44665)
I20260810 23:26:02.668649 24590 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.175.67@KRBTEST.COM'} at 127.14.175.67:44557
I20260810 23:26:02.670123 26652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:02.616785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36278 (local address 127.14.175.124:42991)
0810 23:26:02.620938 (+ 4153us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:02.620944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:02.641964 (+ 21020us) server_negotiation.cc:407] Connection header received
0810 23:26:02.642197 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:02.642203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:02.642331 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:02.642449 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:02.644822 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.646182 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.650299 (+ 4117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.651171 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.651287 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:02.661118 (+ 9831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:02.661128 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:02.661133 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:02.661455 (+ 322us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:02.666017 (+ 4562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:02.667083 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:02.667089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:02.667093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:02.667321 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:02.667889 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:02.667894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:02.667897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:02.668362 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:02.668616 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:02.669347 (+ 731us) server_negotiation.cc:299] Negotiation successful
0810 23:26:02.669656 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4040,"thread_start_us":15601,"threads_started":1}
I20260810 23:26:02.671226 26672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:02.598294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42236 (local address 127.14.175.125:32771)
0810 23:26:02.638366 (+ 40072us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:02.638373 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:02.638395 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:26:02.638454 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:02.638460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:02.638534 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:02.638692 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:02.642939 (+ 4247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.644394 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.650080 (+ 5686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.651200 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.651320 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:02.662363 (+ 11043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:02.662371 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:02.662376 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:02.662416 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:02.666895 (+ 4479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:02.667938 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:02.667944 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:02.667947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:02.668140 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:02.668588 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:02.668593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:02.668596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:02.668969 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:02.669258 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:02.670431 (+ 1173us) server_negotiation.cc:299] Negotiation successful
0810 23:26:02.670679 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39879,"thread_start_us":39688,"threads_started":1}
I20260810 23:26:02.673276 26670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:02.597935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36770 (local address 127.14.175.126:46435)
0810 23:26:02.638815 (+ 40880us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:02.638821 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:02.638840 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:26:02.638892 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:02.638897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:02.638964 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:02.639073 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:02.650063 (+ 10990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.652303 (+ 2240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.655545 (+ 3242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.656621 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.656770 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:02.663734 (+ 6964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:02.663742 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:02.663747 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:02.663786 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:02.669381 (+ 5595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:02.670459 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:02.670465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:02.670468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:02.670722 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:02.671280 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:02.671285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:02.671288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:02.671765 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:02.672063 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:02.672514 (+ 451us) server_negotiation.cc:299] Negotiation successful
0810 23:26:02.672778 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":501,"server-negotiator.queue_time_us":40718,"thread_start_us":39054,"threads_started":1}
I20260810 23:26:02.677793 24590 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:26:02.688679 24590 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:26:02.757136 26704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:02.716845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38090 (local address 127.14.175.67:44665)
0810 23:26:02.731283 (+ 14438us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:02.731296 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:02.734694 (+ 3398us) server_negotiation.cc:407] Connection header received
0810 23:26:02.734848 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:02.734858 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:02.735273 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:02.735630 (+ 357us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:26:02.738152 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.741198 (+ 3046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.751874 (+ 10676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.754514 (+ 2640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.754661 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:02.754943 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:02.756321 (+ 1378us) server_negotiation.cc:299] Negotiation successful
0810 23:26:02.756592 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14109,"thread_start_us":14452,"threads_started":1}
I20260810 23:26:02.758209 26705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:02.716437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:44665 (local address 127.0.0.1:38090)
0810 23:26:02.734213 (+ 17776us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:02.734242 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:02.734514 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:02.736340 (+ 1826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:02.736346 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:02.736386 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:26:02.737778 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:02.737802 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.743963 (+ 6161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.743969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:02.751594 (+ 7625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:02.751611 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.755197 (+ 3586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.755206 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:02.755501 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:02.755590 (+ 89us) client_negotiation.cc:769] Sending connection context
0810 23:26:02.755683 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 23:26:02.755848 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17528,"mutex_wait_us":1,"thread_start_us":17268,"threads_started":1}
I20260810 23:26:02.759341 26699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:02.715895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32922 (local address 127.14.175.66:36553)
0810 23:26:02.743875 (+ 27980us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:02.743885 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:02.743921 (+ 36us) server_negotiation.cc:407] Connection header received
0810 23:26:02.744047 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:02.744056 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:02.744435 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:02.744846 (+ 411us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:26:02.748979 (+ 4133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.750402 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.754356 (+ 3954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.756583 (+ 2227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.756722 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:02.756993 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:02.758699 (+ 1706us) server_negotiation.cc:299] Negotiation successful
0810 23:26:02.758956 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27774,"thread_start_us":27646,"threads_started":1}
I20260810 23:26:02.759827 26698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:02.715619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:36553 (local address 127.0.0.1:32922)
0810 23:26:02.731075 (+ 15456us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:02.731673 (+ 598us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:02.731944 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:02.744851 (+ 12907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:02.744857 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:02.744899 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:26:02.748572 (+ 3673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:02.748591 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.750561 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.750567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:02.754083 (+ 3516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:02.754101 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.757715 (+ 3614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.757728 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:02.758458 (+ 730us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:02.758490 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 23:26:02.758584 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 23:26:02.758736 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"thread_start_us":1363,"threads_started":1}
I20260810 23:26:02.765398 26296 tablet_service.cc:1467] Processing CreateTablet for tablet 7fcb33b6e9e84677806c50f3676f9870 (DEFAULT_TABLE table=test-table [id=3f50278d6a6149398813481a7a48fb48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:26:02.767005 26296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fcb33b6e9e84677806c50f3676f9870. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:26:02.769522 26526 tablet_service.cc:1467] Processing CreateTablet for tablet 7fcb33b6e9e84677806c50f3676f9870 (DEFAULT_TABLE table=test-table [id=3f50278d6a6149398813481a7a48fb48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:26:02.772070 26526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fcb33b6e9e84677806c50f3676f9870. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:26:02.779399 26703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:02.716520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:35079 (local address 127.0.0.1:56920)
0810 23:26:02.742567 (+ 26047us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:02.742593 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:02.742865 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:02.746545 (+ 3680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:02.746550 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:02.746580 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:26:02.750468 (+ 3888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:02.750486 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.753061 (+ 2575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.753067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:02.758315 (+ 5248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:02.758336 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.777714 (+ 19378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.777744 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:02.778106 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:02.778150 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 23:26:02.778295 (+ 145us) client_negotiation.cc:240] Negotiation successful
0810 23:26:02.778474 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25805,"mutex_wait_us":233,"thread_start_us":25148,"threads_started":1}
I20260810 23:26:02.779359 26707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:02.717011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56920 (local address 127.14.175.65:35079)
0810 23:26:02.745848 (+ 28837us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:02.745855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:02.745883 (+ 28us) server_negotiation.cc:407] Connection header received
0810 23:26:02.745966 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:02.745973 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:02.746285 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:02.746548 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:26:02.750740 (+ 4192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.752827 (+ 2087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.758540 (+ 5713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.760923 (+ 2383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.777264 (+ 16341us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:02.777782 (+ 518us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:02.778438 (+ 656us) server_negotiation.cc:299] Negotiation successful
0810 23:26:02.778755 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28604,"thread_start_us":28400,"threads_started":1}
I20260810 23:26:02.785070 26064 tablet_service.cc:1467] Processing CreateTablet for tablet 7fcb33b6e9e84677806c50f3676f9870 (DEFAULT_TABLE table=test-table [id=3f50278d6a6149398813481a7a48fb48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:26:02.786662 26064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fcb33b6e9e84677806c50f3676f9870. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:26:02.793860 26709 tablet_bootstrap.cc:492] T 7fcb33b6e9e84677806c50f3676f9870 P 3892f3eb6ebb436d8bcd0f31647a4c6a: Bootstrap starting.
I20260810 23:26:02.797484 26710 tablet_bootstrap.cc:492] T 7fcb33b6e9e84677806c50f3676f9870 P 28dabb1e621d4aedbd18cb5ff0091cea: Bootstrap starting.
I20260810 23:26:02.800076 26709 tablet_bootstrap.cc:654] T 7fcb33b6e9e84677806c50f3676f9870 P 3892f3eb6ebb436d8bcd0f31647a4c6a: Neither blocks nor log segments found. Creating new log.
I20260810 23:26:02.802738 26710 tablet_bootstrap.cc:654] T 7fcb33b6e9e84677806c50f3676f9870 P 28dabb1e621d4aedbd18cb5ff0091cea: Neither blocks nor log segments found. Creating new log.
I20260810 23:26:02.803086 26709 log.cc:824] T 7fcb33b6e9e84677806c50f3676f9870 P 3892f3eb6ebb436d8bcd0f31647a4c6a: Log is configured to *not* fsync() on all Append() calls
I20260810 23:26:02.805420 26710 log.cc:824] T 7fcb33b6e9e84677806c50f3676f9870 P 28dabb1e621d4aedbd18cb5ff0091cea: Log is configured to *not* fsync() on all Append() calls
I20260810 23:26:02.808254 26709 tablet_bootstrap.cc:492] T 7fcb33b6e9e84677806c50f3676f9870 P 3892f3eb6ebb436d8bcd0f31647a4c6a: No bootstrap required, opened a new log
I20260810 23:26:02.808486 26715 tablet_bootstrap.cc:492] T 7fcb33b6e9e84677806c50f3676f9870 P 4fd367b2b2444451b116d2d69f690b77: Bootstrap starting.
I20260810 23:26:02.808946 26709 ts_tablet_manager.cc:1397] T 7fcb33b6e9e84677806c50f3676f9870 P 3892f3eb6ebb436d8bcd0f31647a4c6a: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.009s
I20260810 23:26:02.811280 26710 tablet_bootstrap.cc:492] T 7fcb33b6e9e84677806c50f3676f9870 P 28dabb1e621d4aedbd18cb5ff0091cea: No bootstrap required, opened a new log
I20260810 23:26:02.811717 26710 ts_tablet_manager.cc:1397] T 7fcb33b6e9e84677806c50f3676f9870 P 28dabb1e621d4aedbd18cb5ff0091cea: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20260810 23:26:02.815132 26715 tablet_bootstrap.cc:654] T 7fcb33b6e9e84677806c50f3676f9870 P 4fd367b2b2444451b116d2d69f690b77: Neither blocks nor log segments found. Creating new log.
I20260810 23:26:02.818549 26715 log.cc:824] T 7fcb33b6e9e84677806c50f3676f9870 P 4fd367b2b2444451b116d2d69f690b77: Log is configured to *not* fsync() on all Append() calls
I20260810 23:26:02.821869 26709 raft_consensus.cc:348] T 7fcb33b6e9e84677806c50f3676f9870 P 3892f3eb6ebb436d8bcd0f31647a4c6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3892f3eb6ebb436d8bcd0f31647a4c6a" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36553 } } peers { permanent_uuid: "28dabb1e621d4aedbd18cb5ff0091cea" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44665 } } peers { permanent_uuid: "4fd367b2b2444451b116d2d69f690b77" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35079 } }
I20260810 23:26:02.822629 26709 raft_consensus.cc:374] T 7fcb33b6e9e84677806c50f3676f9870 P 3892f3eb6ebb436d8bcd0f31647a4c6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:26:02.822800 26709 raft_consensus.cc:729] T 7fcb33b6e9e84677806c50f3676f9870 P 3892f3eb6ebb436d8bcd0f31647a4c6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3892f3eb6ebb436d8bcd0f31647a4c6a, State: Initialized, Role: FOLLOWER
I20260810 23:26:02.823544 26715 tablet_bootstrap.cc:492] T 7fcb33b6e9e84677806c50f3676f9870 P 4fd367b2b2444451b116d2d69f690b77: No bootstrap required, opened a new log
I20260810 23:26:02.824035 26715 ts_tablet_manager.cc:1397] T 7fcb33b6e9e84677806c50f3676f9870 P 4fd367b2b2444451b116d2d69f690b77: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.000s
I20260810 23:26:02.823720 26709 consensus_queue.cc:260] T 7fcb33b6e9e84677806c50f3676f9870 P 3892f3eb6ebb436d8bcd0f31647a4c6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3892f3eb6ebb436d8bcd0f31647a4c6a" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36553 } } peers { permanent_uuid: "28dabb1e621d4aedbd18cb5ff0091cea" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44665 } } peers { permanent_uuid: "4fd367b2b2444451b116d2d69f690b77" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35079 } }
I20260810 23:26:02.825698 26408 heartbeater.cc:503] Master 127.14.175.126:46435 was elected leader, sending a full tablet report...
I20260810 23:26:02.826521 26709 ts_tablet_manager.cc:1428] T 7fcb33b6e9e84677806c50f3676f9870 P 3892f3eb6ebb436d8bcd0f31647a4c6a: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260810 23:26:02.827095 26710 raft_consensus.cc:348] T 7fcb33b6e9e84677806c50f3676f9870 P 28dabb1e621d4aedbd18cb5ff0091cea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3892f3eb6ebb436d8bcd0f31647a4c6a" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36553 } } peers { permanent_uuid: "28dabb1e621d4aedbd18cb5ff0091cea" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44665 } } peers { permanent_uuid: "4fd367b2b2444451b116d2d69f690b77" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35079 } }
I20260810 23:26:02.827867 26710 raft_consensus.cc:374] T 7fcb33b6e9e84677806c50f3676f9870 P 28dabb1e621d4aedbd18cb5ff0091cea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:26:02.828038 26710 raft_consensus.cc:729] T 7fcb33b6e9e84677806c50f3676f9870 P 28dabb1e621d4aedbd18cb5ff0091cea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28dabb1e621d4aedbd18cb5ff0091cea, State: Initialized, Role: FOLLOWER
I20260810 23:26:02.829226 26710 consensus_queue.cc:260] T 7fcb33b6e9e84677806c50f3676f9870 P 28dabb1e621d4aedbd18cb5ff0091cea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3892f3eb6ebb436d8bcd0f31647a4c6a" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36553 } } peers { permanent_uuid: "28dabb1e621d4aedbd18cb5ff0091cea" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44665 } } peers { permanent_uuid: "4fd367b2b2444451b116d2d69f690b77" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35079 } }
I20260810 23:26:02.838887 26710 ts_tablet_manager.cc:1428] T 7fcb33b6e9e84677806c50f3676f9870 P 28dabb1e621d4aedbd18cb5ff0091cea: Time spent starting tablet: real 0.027s user 0.023s sys 0.004s
I20260810 23:26:02.839529 26645 heartbeater.cc:503] Master 127.14.175.126:46435 was elected leader, sending a full tablet report...
I20260810 23:26:02.841655 26715 raft_consensus.cc:348] T 7fcb33b6e9e84677806c50f3676f9870 P 4fd367b2b2444451b116d2d69f690b77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3892f3eb6ebb436d8bcd0f31647a4c6a" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36553 } } peers { permanent_uuid: "28dabb1e621d4aedbd18cb5ff0091cea" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44665 } } peers { permanent_uuid: "4fd367b2b2444451b116d2d69f690b77" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35079 } }
I20260810 23:26:02.842434 26715 raft_consensus.cc:374] T 7fcb33b6e9e84677806c50f3676f9870 P 4fd367b2b2444451b116d2d69f690b77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:26:02.842644 26715 raft_consensus.cc:729] T 7fcb33b6e9e84677806c50f3676f9870 P 4fd367b2b2444451b116d2d69f690b77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd367b2b2444451b116d2d69f690b77, State: Initialized, Role: FOLLOWER
I20260810 23:26:02.843669 26715 consensus_queue.cc:260] T 7fcb33b6e9e84677806c50f3676f9870 P 4fd367b2b2444451b116d2d69f690b77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3892f3eb6ebb436d8bcd0f31647a4c6a" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36553 } } peers { permanent_uuid: "28dabb1e621d4aedbd18cb5ff0091cea" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44665 } } peers { permanent_uuid: "4fd367b2b2444451b116d2d69f690b77" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35079 } }
W20260810 23:26:02.844648 26649 tablet.cc:2369] T 7fcb33b6e9e84677806c50f3676f9870 P 28dabb1e621d4aedbd18cb5ff0091cea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:26:02.846556 26715 ts_tablet_manager.cc:1428] T 7fcb33b6e9e84677806c50f3676f9870 P 4fd367b2b2444451b116d2d69f690b77: Time spent starting tablet: real 0.022s user 0.019s sys 0.004s
I20260810 23:26:02.933039 26722 raft_consensus.cc:482] T 7fcb33b6e9e84677806c50f3676f9870 P 4fd367b2b2444451b116d2d69f690b77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:26:02.933610 26722 raft_consensus.cc:504] T 7fcb33b6e9e84677806c50f3676f9870 P 4fd367b2b2444451b116d2d69f690b77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3892f3eb6ebb436d8bcd0f31647a4c6a" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36553 } } peers { permanent_uuid: "28dabb1e621d4aedbd18cb5ff0091cea" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44665 } } peers { permanent_uuid: "4fd367b2b2444451b116d2d69f690b77" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35079 } }
W20260810 23:26:02.934227 26410 tablet.cc:2369] T 7fcb33b6e9e84677806c50f3676f9870 P 3892f3eb6ebb436d8bcd0f31647a4c6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:26:02.936489 26722 leader_election.cc:288] T 7fcb33b6e9e84677806c50f3676f9870 P 4fd367b2b2444451b116d2d69f690b77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3892f3eb6ebb436d8bcd0f31647a4c6a (127.14.175.66:36553), 28dabb1e621d4aedbd18cb5ff0091cea (127.14.175.67:44665)
I20260810 23:26:02.949709 26704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:02.937242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:56129 (local address 127.14.175.67:44665)
0810 23:26:02.937916 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:02.937922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:02.938881 (+ 959us) server_negotiation.cc:407] Connection header received
0810 23:26:02.939144 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:02.939150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:02.939254 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:02.939341 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:26:02.940943 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.942367 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.946634 (+ 4267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.948340 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.948562 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:02.948643 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:02.949303 (+ 660us) server_negotiation.cc:299] Negotiation successful
0810 23:26:02.949414 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502}
I20260810 23:26:02.949811 26699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:02.937306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:35015 (local address 127.14.175.66:36553)
0810 23:26:02.937802 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:02.937809 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:02.939098 (+ 1289us) server_negotiation.cc:407] Connection header received
0810 23:26:02.939375 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:02.939381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:02.939454 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:02.939538 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:26:02.941072 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.942195 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.946360 (+ 4165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.948359 (+ 1999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.948507 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:02.948586 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:02.949352 (+ 766us) server_negotiation.cc:299] Negotiation successful
0810 23:26:02.949488 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319}
I20260810 23:26:02.950311 26724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:02.936624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:44665 (local address 127.14.175.65:56129)
0810 23:26:02.938720 (+ 2096us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:02.938743 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:02.938951 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:02.939581 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:02.939586 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:02.939624 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:26:02.940754 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:02.940771 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.942548 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.942553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:02.946387 (+ 3834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:02.946401 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.948738 (+ 2337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.948747 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:02.949091 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:02.949133 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 23:26:02.949266 (+ 133us) client_negotiation.cc:240] Negotiation successful
0810 23:26:02.949417 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1929,"mutex_wait_us":88,"spinlock_wait_cycles":1280,"thread_start_us":917,"threads_started":1}
I20260810 23:26:02.950929 26727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:02.937355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:36553 (local address 127.14.175.65:35015)
0810 23:26:02.938875 (+ 1520us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:02.938896 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:02.939216 (+ 320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:02.939581 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:02.939586 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:02.939632 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:26:02.940754 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:02.940771 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.942352 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.942358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:02.946158 (+ 3800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:02.946180 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:02.948887 (+ 2707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:02.948894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:02.949174 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:02.949202 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 23:26:02.950029 (+ 827us) client_negotiation.cc:240] Negotiation successful
0810 23:26:02.950192 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1324,"mutex_wait_us":86,"spinlock_wait_cycles":5632,"thread_start_us":983,"threads_started":1}
I20260810 23:26:02.951977 26577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcb33b6e9e84677806c50f3676f9870" candidate_uuid: "4fd367b2b2444451b116d2d69f690b77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28dabb1e621d4aedbd18cb5ff0091cea" is_pre_election: true
I20260810 23:26:02.952363 26327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcb33b6e9e84677806c50f3676f9870" candidate_uuid: "4fd367b2b2444451b116d2d69f690b77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3892f3eb6ebb436d8bcd0f31647a4c6a" is_pre_election: true
I20260810 23:26:02.952919 26577 raft_consensus.cc:2454] T 7fcb33b6e9e84677806c50f3676f9870 P 28dabb1e621d4aedbd18cb5ff0091cea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd367b2b2444451b116d2d69f690b77 in term 0.
I20260810 23:26:02.953014 26327 raft_consensus.cc:2454] T 7fcb33b6e9e84677806c50f3676f9870 P 3892f3eb6ebb436d8bcd0f31647a4c6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd367b2b2444451b116d2d69f690b77 in term 0.
I20260810 23:26:02.953924 25981 leader_election.cc:302] T 7fcb33b6e9e84677806c50f3676f9870 P 4fd367b2b2444451b116d2d69f690b77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 28dabb1e621d4aedbd18cb5ff0091cea, 3892f3eb6ebb436d8bcd0f31647a4c6a, 4fd367b2b2444451b116d2d69f690b77; no voters:
I20260810 23:26:02.954638 26722 raft_consensus.cc:2781] T 7fcb33b6e9e84677806c50f3676f9870 P 4fd367b2b2444451b116d2d69f690b77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:26:02.954946 26722 raft_consensus.cc:482] T 7fcb33b6e9e84677806c50f3676f9870 P 4fd367b2b2444451b116d2d69f690b77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:26:02.955137 26722 raft_consensus.cc:3037] T 7fcb33b6e9e84677806c50f3676f9870 P 4fd367b2b2444451b116d2d69f690b77 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:26:02.958530 26722 raft_consensus.cc:504] T 7fcb33b6e9e84677806c50f3676f9870 P 4fd367b2b2444451b116d2d69f690b77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3892f3eb6ebb436d8bcd0f31647a4c6a" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36553 } } peers { permanent_uuid: "28dabb1e621d4aedbd18cb5ff0091cea" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44665 } } peers { permanent_uuid: "4fd367b2b2444451b116d2d69f690b77" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35079 } }
I20260810 23:26:02.959328 26722 leader_election.cc:288] T 7fcb33b6e9e84677806c50f3676f9870 P 4fd367b2b2444451b116d2d69f690b77 [CANDIDATE]: Term 1 election: Requested vote from peers 3892f3eb6ebb436d8bcd0f31647a4c6a (127.14.175.66:36553), 28dabb1e621d4aedbd18cb5ff0091cea (127.14.175.67:44665)
I20260810 23:26:02.960047 26327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcb33b6e9e84677806c50f3676f9870" candidate_uuid: "4fd367b2b2444451b116d2d69f690b77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3892f3eb6ebb436d8bcd0f31647a4c6a"
I20260810 23:26:02.960072 26577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcb33b6e9e84677806c50f3676f9870" candidate_uuid: "4fd367b2b2444451b116d2d69f690b77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28dabb1e621d4aedbd18cb5ff0091cea"
I20260810 23:26:02.960350 26327 raft_consensus.cc:3037] T 7fcb33b6e9e84677806c50f3676f9870 P 3892f3eb6ebb436d8bcd0f31647a4c6a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:26:02.960398 26577 raft_consensus.cc:3037] T 7fcb33b6e9e84677806c50f3676f9870 P 28dabb1e621d4aedbd18cb5ff0091cea [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:26:02.963912 26577 raft_consensus.cc:2454] T 7fcb33b6e9e84677806c50f3676f9870 P 28dabb1e621d4aedbd18cb5ff0091cea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd367b2b2444451b116d2d69f690b77 in term 1.
I20260810 23:26:02.963941 26327 raft_consensus.cc:2454] T 7fcb33b6e9e84677806c50f3676f9870 P 3892f3eb6ebb436d8bcd0f31647a4c6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd367b2b2444451b116d2d69f690b77 in term 1.
I20260810 23:26:02.964486 25981 leader_election.cc:302] T 7fcb33b6e9e84677806c50f3676f9870 P 4fd367b2b2444451b116d2d69f690b77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28dabb1e621d4aedbd18cb5ff0091cea, 4fd367b2b2444451b116d2d69f690b77; no voters:
I20260810 23:26:02.964906 26722 raft_consensus.cc:2781] T 7fcb33b6e9e84677806c50f3676f9870 P 4fd367b2b2444451b116d2d69f690b77 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:26:02.965898 26722 raft_consensus.cc:686] T 7fcb33b6e9e84677806c50f3676f9870 P 4fd367b2b2444451b116d2d69f690b77 [term 1 LEADER]: Becoming Leader. State: Replica: 4fd367b2b2444451b116d2d69f690b77, State: Running, Role: LEADER
I20260810 23:26:02.966661 26722 consensus_queue.cc:237] T 7fcb33b6e9e84677806c50f3676f9870 P 4fd367b2b2444451b116d2d69f690b77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3892f3eb6ebb436d8bcd0f31647a4c6a" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36553 } } peers { permanent_uuid: "28dabb1e621d4aedbd18cb5ff0091cea" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44665 } } peers { permanent_uuid: "4fd367b2b2444451b116d2d69f690b77" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35079 } }
I20260810 23:26:02.973651 24587 catalog_manager.cc:5495] T 7fcb33b6e9e84677806c50f3676f9870 P 4fd367b2b2444451b116d2d69f690b77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fd367b2b2444451b116d2d69f690b77 (127.14.175.65). New cstate: current_term: 1 leader_uuid: "4fd367b2b2444451b116d2d69f690b77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3892f3eb6ebb436d8bcd0f31647a4c6a" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28dabb1e621d4aedbd18cb5ff0091cea" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd367b2b2444451b116d2d69f690b77" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35079 } health_report { overall_health: HEALTHY } } }
I20260810 23:26:03.015615 26707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:03.007443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56928 (local address 127.14.175.65:35079)
0810 23:26:03.007684 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:03.007690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:03.007742 (+ 52us) server_negotiation.cc:407] Connection header received
0810 23:26:03.008006 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:03.008011 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:03.008127 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:03.008311 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:26:03.008770 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:03.010201 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:03.012041 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:03.012864 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:03.013007 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:03.013960 (+ 953us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:26:03.014732 (+ 772us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:26:03.014906 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:03.015041 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 23:26:03.015161 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 23:26:03.028252 26577 raft_consensus.cc:1261] T 7fcb33b6e9e84677806c50f3676f9870 P 28dabb1e621d4aedbd18cb5ff0091cea [term 1 FOLLOWER]: Refusing update from remote peer 4fd367b2b2444451b116d2d69f690b77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:26:03.028551 26327 raft_consensus.cc:1261] T 7fcb33b6e9e84677806c50f3676f9870 P 3892f3eb6ebb436d8bcd0f31647a4c6a [term 1 FOLLOWER]: Refusing update from remote peer 4fd367b2b2444451b116d2d69f690b77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:26:03.029420 26722 consensus_queue.cc:1035] T 7fcb33b6e9e84677806c50f3676f9870 P 4fd367b2b2444451b116d2d69f690b77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28dabb1e621d4aedbd18cb5ff0091cea" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:26:03.029800 26728 consensus_queue.cc:1035] T 7fcb33b6e9e84677806c50f3676f9870 P 4fd367b2b2444451b116d2d69f690b77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3892f3eb6ebb436d8bcd0f31647a4c6a" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 23:26:03.030596 26175 tablet.cc:2369] T 7fcb33b6e9e84677806c50f3676f9870 P 4fd367b2b2444451b116d2d69f690b77: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:26:03.061296 26741 mvcc.cc:205] Tried to move back new op lower bound from 7317112270951358464 to 7317112270716354560. Current Snapshot: MvccSnapshot[applied={T|T < 7317112270951358464}]
I20260810 23:26:03.084712 24587 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:26:03.085259 24587 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:26:03.092729 24587 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 79ba9107e6d44ecf90ec45f01d3ec450: Sending DeleteTablet for 3 replicas of tablet 7fcb33b6e9e84677806c50f3676f9870
I20260810 23:26:03.094645 26296 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fcb33b6e9e84677806c50f3676f9870 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:26:03 UTC) from {username='oryx', principal='oryx/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:32922
I20260810 23:26:03.095417 26064 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fcb33b6e9e84677806c50f3676f9870 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:26:03 UTC) from {username='oryx', principal='oryx/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:56920
I20260810 23:26:03.095901 26526 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fcb33b6e9e84677806c50f3676f9870 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:26:03 UTC) from {username='oryx', principal='oryx/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:38090
I20260810 23:26:03.100260 26758 tablet_replica.cc:330] T 7fcb33b6e9e84677806c50f3676f9870 P 28dabb1e621d4aedbd18cb5ff0091cea: stopping tablet replica
I20260810 23:26:03.100873 26758 raft_consensus.cc:2229] T 7fcb33b6e9e84677806c50f3676f9870 P 28dabb1e621d4aedbd18cb5ff0091cea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:26:03.102881 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 24540
I20260810 23:26:03.104225 26758 raft_consensus.cc:2258] T 7fcb33b6e9e84677806c50f3676f9870 P 28dabb1e621d4aedbd18cb5ff0091cea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:26:03.104446 24636 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:26:03.108243 26755 tablet_replica.cc:330] T 7fcb33b6e9e84677806c50f3676f9870 P 3892f3eb6ebb436d8bcd0f31647a4c6a: stopping tablet replica
I20260810 23:26:03.108817 26755 raft_consensus.cc:2229] T 7fcb33b6e9e84677806c50f3676f9870 P 3892f3eb6ebb436d8bcd0f31647a4c6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:26:03.109513 26758 ts_tablet_manager.cc:1905] T 7fcb33b6e9e84677806c50f3676f9870 P 28dabb1e621d4aedbd18cb5ff0091cea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:26:03.109899 26755 raft_consensus.cc:2258] T 7fcb33b6e9e84677806c50f3676f9870 P 3892f3eb6ebb436d8bcd0f31647a4c6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:26:03.112416 26755 ts_tablet_manager.cc:1905] T 7fcb33b6e9e84677806c50f3676f9870 P 3892f3eb6ebb436d8bcd0f31647a4c6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:26:03.121671 26757 tablet_replica.cc:330] T 7fcb33b6e9e84677806c50f3676f9870 P 4fd367b2b2444451b116d2d69f690b77: stopping tablet replica
I20260810 23:26:03.122069 26755 ts_tablet_manager.cc:1918] T 7fcb33b6e9e84677806c50f3676f9870 P 3892f3eb6ebb436d8bcd0f31647a4c6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:26:03.122370 26755 log.cc:1196] T 7fcb33b6e9e84677806c50f3676f9870 P 3892f3eb6ebb436d8bcd0f31647a4c6a: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-1/wal/wals/7fcb33b6e9e84677806c50f3676f9870
I20260810 23:26:03.122364 26757 raft_consensus.cc:2229] T 7fcb33b6e9e84677806c50f3676f9870 P 4fd367b2b2444451b116d2d69f690b77 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:26:03.123373 26755 ts_tablet_manager.cc:1939] T 7fcb33b6e9e84677806c50f3676f9870 P 3892f3eb6ebb436d8bcd0f31647a4c6a: Deleting consensus metadata
I20260810 23:26:03.123464 26757 raft_consensus.cc:2258] T 7fcb33b6e9e84677806c50f3676f9870 P 4fd367b2b2444451b116d2d69f690b77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:26:03.128907 26758 ts_tablet_manager.cc:1918] T 7fcb33b6e9e84677806c50f3676f9870 P 28dabb1e621d4aedbd18cb5ff0091cea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:26:03.129343 26758 log.cc:1196] T 7fcb33b6e9e84677806c50f3676f9870 P 28dabb1e621d4aedbd18cb5ff0091cea: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-2/wal/wals/7fcb33b6e9e84677806c50f3676f9870
I20260810 23:26:03.130288 26758 ts_tablet_manager.cc:1939] T 7fcb33b6e9e84677806c50f3676f9870 P 28dabb1e621d4aedbd18cb5ff0091cea: Deleting consensus metadata
I20260810 23:26:03.133641 26757 ts_tablet_manager.cc:1905] T 7fcb33b6e9e84677806c50f3676f9870 P 4fd367b2b2444451b116d2d69f690b77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:26:03.135162 26672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:03.097436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42246 (local address 127.14.175.125:32771)
0810 23:26:03.097749 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:03.097756 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:03.102933 (+ 5177us) server_negotiation.cc:407] Connection header received
0810 23:26:03.103028 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:03.103035 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:03.103144 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:03.103259 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:03.110464 (+ 7205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:03.112040 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:03.115212 (+ 3172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:03.116344 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:03.116460 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:03.123238 (+ 6778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:03.123249 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:03.123254 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:03.123310 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:03.129295 (+ 5985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:03.130750 (+ 1455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:03.130758 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:03.130762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:03.131111 (+ 349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:03.131709 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:03.131715 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:03.131718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:03.132273 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:03.132646 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:03.133514 (+ 868us) server_negotiation.cc:299] Negotiation successful
0810 23:26:03.133813 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 23:26:03.145226 26757 ts_tablet_manager.cc:1918] T 7fcb33b6e9e84677806c50f3676f9870 P 4fd367b2b2444451b116d2d69f690b77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:26:03.145681 26757 log.cc:1196] T 7fcb33b6e9e84677806c50f3676f9870 P 4fd367b2b2444451b116d2d69f690b77: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-0/wal/wals/7fcb33b6e9e84677806c50f3676f9870
I20260810 23:26:03.146831 26757 ts_tablet_manager.cc:1939] T 7fcb33b6e9e84677806c50f3676f9870 P 4fd367b2b2444451b116d2d69f690b77: Deleting consensus metadata
I20260810 23:26:03.160327 26652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:03.102670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36290 (local address 127.14.175.124:42991)
0810 23:26:03.102868 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:03.102875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:03.134471 (+ 31596us) server_negotiation.cc:407] Connection header received
0810 23:26:03.134628 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:03.134634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:03.134705 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:03.134789 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:03.140740 (+ 5951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:03.141762 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:03.143918 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:03.144538 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:03.144676 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:03.150382 (+ 5706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:03.150391 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:03.150396 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:03.150451 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:03.156144 (+ 5693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:03.157227 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:03.157234 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:03.157238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:03.157503 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:03.158078 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:03.158085 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:03.158088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:03.158648 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:03.158959 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:03.159553 (+ 594us) server_negotiation.cc:299] Negotiation successful
0810 23:26:03.159846 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 23:26:03.272330 24564 catalog_manager.cc:4841] TS 4fd367b2b2444451b116d2d69f690b77 (127.14.175.65:35079): tablet 7fcb33b6e9e84677806c50f3676f9870 (table test-table [id=3f50278d6a6149398813481a7a48fb48]) successfully deleted
I20260810 23:26:03.272734 24565 catalog_manager.cc:4841] TS 28dabb1e621d4aedbd18cb5ff0091cea (127.14.175.67:44665): tablet 7fcb33b6e9e84677806c50f3676f9870 (table test-table [id=3f50278d6a6149398813481a7a48fb48]) successfully deleted
I20260810 23:26:03.273211 24567 catalog_manager.cc:4841] TS 3892f3eb6ebb436d8bcd0f31647a4c6a (127.14.175.66:36553): tablet 7fcb33b6e9e84677806c50f3676f9870 (table test-table [id=3f50278d6a6149398813481a7a48fb48]) successfully deleted
I20260810 23:26:03.273995 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 24540
I20260810 23:26:03.328943 26672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:03.310305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42248 (local address 127.14.175.125:32771)
0810 23:26:03.310520 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:03.310527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:03.311167 (+ 640us) server_negotiation.cc:407] Connection header received
0810 23:26:03.311392 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:03.311399 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:03.311501 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:03.311620 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:03.314291 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:03.315659 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:03.317002 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:03.317662 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:03.317797 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:03.321698 (+ 3901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:03.321707 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:03.321713 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:03.321768 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:03.325049 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:03.326283 (+ 1234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:03.326290 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:03.326294 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:03.326462 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:03.326974 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:03.326979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:03.326981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:03.327408 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:03.327666 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:03.328353 (+ 687us) server_negotiation.cc:299] Negotiation successful
0810 23:26:03.328626 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 23:26:03.333674 26652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:03.312643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36292 (local address 127.14.175.124:42991)
0810 23:26:03.312947 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:03.312954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:03.312977 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:26:03.313044 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:03.313049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:03.313130 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:03.313283 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:03.315747 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:03.316942 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:03.318819 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:03.319551 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:03.319692 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:03.323306 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:03.323314 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:03.323320 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:03.323378 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:03.326954 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:03.328059 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:03.328065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:03.328068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:03.328369 (+ 301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:03.328805 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:03.328811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:03.328814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:03.331987 (+ 3173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:03.332361 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:03.332722 (+ 361us) server_negotiation.cc:299] Negotiation successful
0810 23:26:03.333021 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 23:26:04.072037 26777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:04.071160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.14.175.67:58011)
0810 23:26:04.071694 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:04.071823 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":258,"threads_started":1}
W20260810 23:26:04.072623 26645 heartbeater.cc:644] Failed to heartbeat to 127.14.175.126:46435 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
I20260810 23:26:04.076349 26778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:04.075213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.14.175.66:55305)
0810 23:26:04.076057 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:04.076189 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":596,"threads_started":1}
W20260810 23:26:04.076761 26408 heartbeater.cc:644] Failed to heartbeat to 127.14.175.126:46435 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
I20260810 23:26:04.079532 26779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:04.078198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.14.175.65:59343)
0810 23:26:04.079158 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:04.079316 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":844,"thread_start_us":730,"threads_started":1}
W20260810 23:26:04.079941 26173 heartbeater.cc:644] Failed to heartbeat to 127.14.175.126:46435 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
I20260810 23:26:04.593528 26782 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 79ba9107e6d44ecf90ec45f01d3ec450)
I20260810 23:26:04.593811 26782 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: false } }
I20260810 23:26:04.594223 26781 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 79ba9107e6d44ecf90ec45f01d3ec450)
I20260810 23:26:04.594408 26781 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: false } }
I20260810 23:26:04.594692 26782 leader_election.cc:288] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 79ba9107e6d44ecf90ec45f01d3ec450 (127.14.175.126:46435), 23fb6044cc6444828df1c91f64d20796 (127.14.175.124:42991)
I20260810 23:26:04.595288 26781 leader_election.cc:288] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 79ba9107e6d44ecf90ec45f01d3ec450 (127.14.175.126:46435), c8e7587e7a104723916c49843b278e0f (127.14.175.125:32771)
I20260810 23:26:04.602185 26783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:04.594928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.0.0.1:37194)
0810 23:26:04.595510 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:04.595766 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":216,"threads_started":1}
I20260810 23:26:04.607331 26787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:04.595437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.0.0.1:37206)
0810 23:26:04.606760 (+ 11323us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:04.607007 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11151,"threads_started":1}
W20260810 23:26:04.608606 25851 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
W20260810 23:26:04.608776 25522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
I20260810 23:26:04.616191 26783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:04.595591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:42991 (local address 127.0.0.1:36306)
0810 23:26:04.602976 (+ 7385us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:04.603003 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:04.603190 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:04.608915 (+ 5725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:04.608921 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:04.609064 (+ 143us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:26:04.609340 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:04.609353 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:04.611395 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:04.611402 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:04.613663 (+ 2261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:04.613681 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:04.615455 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:04.615461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:04.615642 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:04.615678 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 23:26:04.615755 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 23:26:04.615903 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7259,"thread_start_us":12102,"threads_started":1}
I20260810 23:26:04.616318 26787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:04.595657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:32771 (local address 127.0.0.1:42252)
0810 23:26:04.607556 (+ 11899us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:04.607578 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:04.607746 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:04.609126 (+ 1380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:04.609131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:04.609196 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:26:04.609445 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:04.609458 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:04.611413 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:04.611420 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:04.613680 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:04.613699 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:04.615485 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:04.615492 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:04.615707 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:04.615751 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 23:26:04.615818 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 23:26:04.615969 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11803,"thread_start_us":12174,"threads_started":1}
I20260810 23:26:04.616736 26784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:04.616160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.0.0.1:37212)
0810 23:26:04.616326 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:04.616535 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 23:26:04.617008 25851 leader_election.cc:334] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 79ba9107e6d44ecf90ec45f01d3ec450 (127.14.175.126:46435): Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
I20260810 23:26:04.617508 26790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:04.607967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42252 (local address 127.14.175.125:32771)
0810 23:26:04.608781 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:04.608786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:04.608807 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:26:04.608872 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:04.608876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:04.608944 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:04.609596 (+ 652us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:26:04.609873 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:04.611213 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:04.613910 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:04.615266 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:04.616753 (+ 1487us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:04.616833 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:04.616887 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 23:26:04.617027 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":511,"threads_started":1}
I20260810 23:26:04.617807 26789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:04.596058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36306 (local address 127.14.175.124:42991)
0810 23:26:04.608432 (+ 12374us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:04.608438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:04.608458 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:26:04.608518 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:04.608522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:04.608599 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:04.609494 (+ 895us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:26:04.609833 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:04.611187 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:04.613898 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:04.615233 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:04.616947 (+ 1714us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:04.617019 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:04.617083 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 23:26:04.617307 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12224,"thread_start_us":12645,"threads_started":1}
I20260810 23:26:04.618088 25561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "23fb6044cc6444828df1c91f64d20796" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c8e7587e7a104723916c49843b278e0f" is_pre_election: true
I20260810 23:26:04.618345 25561 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23fb6044cc6444828df1c91f64d20796 in term 1.
I20260810 23:26:04.619082 25856 leader_election.cc:302] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23fb6044cc6444828df1c91f64d20796, c8e7587e7a104723916c49843b278e0f; no voters: 79ba9107e6d44ecf90ec45f01d3ec450
I20260810 23:26:04.619181 26783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:04.618738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.0.0.1:37222)
0810 23:26:04.618906 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:04.619047 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 23:26:04.619033 25892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8e7587e7a104723916c49843b278e0f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "23fb6044cc6444828df1c91f64d20796" is_pre_election: true
W20260810 23:26:04.619369 25522 leader_election.cc:334] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 79ba9107e6d44ecf90ec45f01d3ec450 (127.14.175.126:46435): Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
I20260810 23:26:04.619421 25892 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8e7587e7a104723916c49843b278e0f in term 1.
I20260810 23:26:04.619623 26781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 23:26:04.619751 26781 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 79ba9107e6d44ecf90ec45f01d3ec450)
I20260810 23:26:04.619858 26781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:26:04.619906 25522 leader_election.cc:302] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23fb6044cc6444828df1c91f64d20796, c8e7587e7a104723916c49843b278e0f; no voters: 79ba9107e6d44ecf90ec45f01d3ec450
I20260810 23:26:04.620204 26782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 23:26:04.620328 26782 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 79ba9107e6d44ecf90ec45f01d3ec450)
I20260810 23:26:04.620414 26782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:26:04.622212 26781 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: false } }
I20260810 23:26:04.622605 26782 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: false } }
I20260810 23:26:04.622818 26781 leader_election.cc:288] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [CANDIDATE]: Term 2 election: Requested vote from peers 79ba9107e6d44ecf90ec45f01d3ec450 (127.14.175.126:46435), c8e7587e7a104723916c49843b278e0f (127.14.175.125:32771)
I20260810 23:26:04.623306 26782 leader_election.cc:288] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [CANDIDATE]: Term 2 election: Requested vote from peers 79ba9107e6d44ecf90ec45f01d3ec450 (127.14.175.126:46435), 23fb6044cc6444828df1c91f64d20796 (127.14.175.124:42991)
I20260810 23:26:04.623391 26784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:04.623021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.0.0.1:37236)
0810 23:26:04.623170 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:04.623255 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 23:26:04.624069 25892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8e7587e7a104723916c49843b278e0f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "23fb6044cc6444828df1c91f64d20796"
I20260810 23:26:04.624364 25892 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c8e7587e7a104723916c49843b278e0f in current term 2: Already voted for candidate 23fb6044cc6444828df1c91f64d20796 in this term.
I20260810 23:26:04.624497 25561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "23fb6044cc6444828df1c91f64d20796" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c8e7587e7a104723916c49843b278e0f"
I20260810 23:26:04.624816 25561 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 23fb6044cc6444828df1c91f64d20796 in current term 2: Already voted for candidate c8e7587e7a104723916c49843b278e0f in this term.
I20260810 23:26:04.625341 26783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:04.623451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.0.0.1:37254)
0810 23:26:04.625127 (+ 1676us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:04.625232 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1576}
I20260810 23:26:04.625432 26784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:04.625049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.0.0.1:37258)
0810 23:26:04.625215 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:04.625296 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 23:26:04.625672 25851 leader_election.cc:334] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 79ba9107e6d44ecf90ec45f01d3ec450 (127.14.175.126:46435): Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
I20260810 23:26:04.625846 25851 leader_election.cc:302] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23fb6044cc6444828df1c91f64d20796; no voters: 79ba9107e6d44ecf90ec45f01d3ec450, c8e7587e7a104723916c49843b278e0f
I20260810 23:26:04.626122 26781 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 23:26:04.626240 26783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:04.625889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.0.0.1:37266)
0810 23:26:04.626053 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:04.626120 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 23:26:04.626380 25522 leader_election.cc:334] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 79ba9107e6d44ecf90ec45f01d3ec450 (127.14.175.126:46435): Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
I20260810 23:26:04.626520 25522 leader_election.cc:302] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8e7587e7a104723916c49843b278e0f; no voters: 23fb6044cc6444828df1c91f64d20796, 79ba9107e6d44ecf90ec45f01d3ec450
I20260810 23:26:04.626784 26782 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 23:26:05.074301 26796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:05.073488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.14.175.67:39155)
0810 23:26:05.073948 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:05.074096 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":237,"threads_started":1}
I20260810 23:26:05.078296 26797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:05.077505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.14.175.66:46221)
0810 23:26:05.078026 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:05.078122 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":302,"threads_started":1}
I20260810 23:26:05.081558 26799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:05.080687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.14.175.65:42553)
0810 23:26:05.081213 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:05.081304 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":289,"threads_started":1}
I20260810 23:26:06.076051 26800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:06.075260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.14.175.67:58331)
0810 23:26:06.075757 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:06.075867 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":277,"threads_started":1}
I20260810 23:26:06.079846 26801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:06.079148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.14.175.66:33931)
0810 23:26:06.079585 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:06.079675 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":231,"threads_started":1}
I20260810 23:26:06.083259 26802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:06.082501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.14.175.65:56613)
0810 23:26:06.082967 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:06.083073 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":257,"threads_started":1}
I20260810 23:26:06.576512 26803 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:26:06.576786 26803 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: false } }
I20260810 23:26:06.577559 26803 leader_election.cc:288] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 79ba9107e6d44ecf90ec45f01d3ec450 (127.14.175.126:46435), 23fb6044cc6444828df1c91f64d20796 (127.14.175.124:42991)
I20260810 23:26:06.578699 26804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:06.577755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.0.0.1:37540)
0810 23:26:06.578248 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:06.578406 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":228,"threads_started":1}
I20260810 23:26:06.578653 25892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8e7587e7a104723916c49843b278e0f" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "23fb6044cc6444828df1c91f64d20796" is_pre_election: true
I20260810 23:26:06.578873 25892 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8e7587e7a104723916c49843b278e0f in term 2.
I20260810 23:26:06.579458 25522 leader_election.cc:302] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23fb6044cc6444828df1c91f64d20796, c8e7587e7a104723916c49843b278e0f; no voters:
I20260810 23:26:06.579767 26803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 23:26:06.579896 26803 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:26:06.580014 26803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 2 FOLLOWER]: Advancing to term 3
I20260810 23:26:06.580296 26804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:06.579906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.0.0.1:37556)
0810 23:26:06.580066 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:06.580168 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 23:26:06.580513 25522 leader_election.cc:334] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 79ba9107e6d44ecf90ec45f01d3ec450 (127.14.175.126:46435): Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
I20260810 23:26:06.582376 26803 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: false } }
I20260810 23:26:06.582990 26803 leader_election.cc:288] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [CANDIDATE]: Term 3 election: Requested vote from peers 79ba9107e6d44ecf90ec45f01d3ec450 (127.14.175.126:46435), 23fb6044cc6444828df1c91f64d20796 (127.14.175.124:42991)
I20260810 23:26:06.583537 26804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:06.583125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.0.0.1:37560)
0810 23:26:06.583271 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:06.583362 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 23:26:06.583775 25892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8e7587e7a104723916c49843b278e0f" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "23fb6044cc6444828df1c91f64d20796"
I20260810 23:26:06.583992 25892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [term 2 FOLLOWER]: Advancing to term 3
I20260810 23:26:06.585108 26804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:06.584701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.0.0.1:37572)
0810 23:26:06.584866 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:06.584953 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260810 23:26:06.585299 25522 leader_election.cc:334] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 79ba9107e6d44ecf90ec45f01d3ec450 (127.14.175.126:46435): Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
I20260810 23:26:06.586468 25892 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8e7587e7a104723916c49843b278e0f in term 3.
I20260810 23:26:06.586908 25522 leader_election.cc:302] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23fb6044cc6444828df1c91f64d20796, c8e7587e7a104723916c49843b278e0f; no voters: 79ba9107e6d44ecf90ec45f01d3ec450
I20260810 23:26:06.587208 26803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 3 FOLLOWER]: Leader election won for term 3
I20260810 23:26:06.587607 26803 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [term 3 LEADER]: Becoming Leader. State: Replica: c8e7587e7a104723916c49843b278e0f, State: Running, Role: LEADER
I20260810 23:26:06.587984 26803 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: false } }
I20260810 23:26:06.589808 26806 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8e7587e7a104723916c49843b278e0f. Latest consensus state: current_term: 3 leader_uuid: "c8e7587e7a104723916c49843b278e0f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: false } } }
I20260810 23:26:06.590219 26806 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [sys.catalog]: This master's current role is: LEADER
I20260810 23:26:06.590987 26811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:26:06.595577 26811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:26:06.596400 26811 catalog_manager.cc:1259] Loaded cluster ID: 8d5f154f21774aaa952ac2fb0c6079bd
I20260810 23:26:06.596473 26811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:26:06.597671 26811 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:26:06.598471 26811 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f: Loaded TSK: 0
I20260810 23:26:06.599388 26811 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:26:06.620796 25546 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:26:06.628084 25546 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:26:06.632956 26804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:06.632485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.0.0.1:37590)
0810 23:26:06.632663 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:06.632795 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 23:26:06.633450 25892 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [term 3 FOLLOWER]: Refusing update from remote peer c8e7587e7a104723916c49843b278e0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260810 23:26:06.634048 26804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:06.633659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.0.0.1:37596)
0810 23:26:06.633842 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:06.633925 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 23:26:06.634225 26806 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260810 23:26:06.634507 25522 consensus_peers.cc:487] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f -> Peer 79ba9107e6d44ecf90ec45f01d3ec450 (127.14.175.126:46435): Couldn't send request to peer 79ba9107e6d44ecf90ec45f01d3ec450. Status: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 23:26:06.636881 26815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8e7587e7a104723916c49843b278e0f. Latest consensus state: current_term: 3 leader_uuid: "c8e7587e7a104723916c49843b278e0f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: false } } }
I20260810 23:26:06.637190 26815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:26:06.638687 26806 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "c8e7587e7a104723916c49843b278e0f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: false } } }
I20260810 23:26:06.638963 26806 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f [sys.catalog]: This master's current role is: LEADER
I20260810 23:26:06.639351 26815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "c8e7587e7a104723916c49843b278e0f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "79ba9107e6d44ecf90ec45f01d3ec450" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46435 } } peers { permanent_uuid: "c8e7587e7a104723916c49843b278e0f" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 32771 } attrs { promote: false } } peers { permanent_uuid: "23fb6044cc6444828df1c91f64d20796" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42991 } attrs { promote: false } } }
I20260810 23:26:06.639597 26815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23fb6044cc6444828df1c91f64d20796 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:26:06.671486 26804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:06.647657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:44665 (local address 127.0.0.1:38102)
0810 23:26:06.647930 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:06.647952 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:06.648078 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:06.666072 (+ 17994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:06.666076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:06.666101 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:26:06.666399 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:06.666411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:06.667860 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:06.667864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:06.669232 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:06.669251 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:06.670663 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:06.670670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:06.670854 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:06.670898 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 23:26:06.670971 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 23:26:06.671133 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260810 23:26:06.672757 26824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:06.647747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38102 (local address 127.14.175.67:44665)
0810 23:26:06.665713 (+ 17966us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:06.665720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:06.665741 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:26:06.665822 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:06.665828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:06.665937 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:06.666044 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:26:06.666560 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:06.667716 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:06.669429 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:06.670408 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:06.671758 (+ 1350us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:06.671874 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:06.672341 (+ 467us) server_negotiation.cc:299] Negotiation successful
0810 23:26:06.672505 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17820,"thread_start_us":17394,"threads_started":1}
I20260810 23:26:06.673652 26526 tablet_service.cc:1467] Processing CreateTablet for tablet f7e41ddb74d04b06b5e706ad44c1482e (DEFAULT_TABLE table=test-table [id=f6654b18c03647aaadd8a3b73c42c3cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:26:06.674163 26526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7e41ddb74d04b06b5e706ad44c1482e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:26:06.676251 26829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:06.647925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:35079 (local address 127.0.0.1:56930)
0810 23:26:06.663662 (+ 15737us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:06.663689 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:06.663862 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:06.669993 (+ 6131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:06.669998 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:06.670033 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:26:06.670531 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:06.670547 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:06.671950 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:06.671955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:06.674049 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:06.674061 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:06.675419 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:06.675426 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:06.675628 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:06.675656 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 23:26:06.675745 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 23:26:06.675899 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15534,"thread_start_us":22400,"threads_started":1}
I20260810 23:26:06.677067 26828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:06.648324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56930 (local address 127.14.175.65:35079)
0810 23:26:06.669624 (+ 21300us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:06.669631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:06.669654 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:26:06.669735 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:06.669741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:06.669833 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:06.669949 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:26:06.670787 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:06.671746 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:06.674231 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:06.675203 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:06.675339 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:06.675443 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:06.676660 (+ 1217us) server_negotiation.cc:299] Negotiation successful
0810 23:26:06.676798 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21140,"thread_start_us":21028,"threads_started":1}
I20260810 23:26:06.677953 26064 tablet_service.cc:1467] Processing CreateTablet for tablet f7e41ddb74d04b06b5e706ad44c1482e (DEFAULT_TABLE table=test-table [id=f6654b18c03647aaadd8a3b73c42c3cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:26:06.678499 26064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7e41ddb74d04b06b5e706ad44c1482e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:26:06.680974 26825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:06.648500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:36553 (local address 127.0.0.1:32936)
0810 23:26:06.670655 (+ 22155us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:06.670673 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:06.670808 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:06.673352 (+ 2544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:06.673358 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:06.673388 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:26:06.673797 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:06.673810 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:06.675244 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:06.675250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:06.677416 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:06.677436 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:06.680059 (+ 2623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:06.680066 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:06.680280 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:06.680320 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 23:26:06.680409 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 23:26:06.680590 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22014,"thread_start_us":16190,"threads_started":1}
I20260810 23:26:06.681056 26831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:06.648680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32936 (local address 127.14.175.66:36553)
0810 23:26:06.664515 (+ 15835us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:06.664523 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:06.672978 (+ 8455us) server_negotiation.cc:407] Connection header received
0810 23:26:06.673074 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:06.673078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:06.673197 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:06.673314 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:26:06.673995 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:06.675044 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:06.677628 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:06.679726 (+ 2098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:06.680286 (+ 560us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:06.680409 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:06.680468 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 23:26:06.680619 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15659,"thread_start_us":15850,"threads_started":1}
I20260810 23:26:06.682102 26833 tablet_bootstrap.cc:492] T f7e41ddb74d04b06b5e706ad44c1482e P 28dabb1e621d4aedbd18cb5ff0091cea: Bootstrap starting.
I20260810 23:26:06.682219 26296 tablet_service.cc:1467] Processing CreateTablet for tablet f7e41ddb74d04b06b5e706ad44c1482e (DEFAULT_TABLE table=test-table [id=f6654b18c03647aaadd8a3b73c42c3cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:26:06.682727 26296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7e41ddb74d04b06b5e706ad44c1482e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:26:06.683800 26833 tablet_bootstrap.cc:654] T f7e41ddb74d04b06b5e706ad44c1482e P 28dabb1e621d4aedbd18cb5ff0091cea: Neither blocks nor log segments found. Creating new log.
I20260810 23:26:06.686100 26833 tablet_bootstrap.cc:492] T f7e41ddb74d04b06b5e706ad44c1482e P 28dabb1e621d4aedbd18cb5ff0091cea: No bootstrap required, opened a new log
I20260810 23:26:06.686208 26835 tablet_bootstrap.cc:492] T f7e41ddb74d04b06b5e706ad44c1482e P 4fd367b2b2444451b116d2d69f690b77: Bootstrap starting.
I20260810 23:26:06.686391 26833 ts_tablet_manager.cc:1397] T f7e41ddb74d04b06b5e706ad44c1482e P 28dabb1e621d4aedbd18cb5ff0091cea: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260810 23:26:06.687271 26833 raft_consensus.cc:348] T f7e41ddb74d04b06b5e706ad44c1482e P 28dabb1e621d4aedbd18cb5ff0091cea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28dabb1e621d4aedbd18cb5ff0091cea" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44665 } } peers { permanent_uuid: "4fd367b2b2444451b116d2d69f690b77" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35079 } } peers { permanent_uuid: "3892f3eb6ebb436d8bcd0f31647a4c6a" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36553 } }
I20260810 23:26:06.687516 26833 raft_consensus.cc:374] T f7e41ddb74d04b06b5e706ad44c1482e P 28dabb1e621d4aedbd18cb5ff0091cea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:26:06.687659 26833 raft_consensus.cc:729] T f7e41ddb74d04b06b5e706ad44c1482e P 28dabb1e621d4aedbd18cb5ff0091cea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28dabb1e621d4aedbd18cb5ff0091cea, State: Initialized, Role: FOLLOWER
I20260810 23:26:06.687935 26833 consensus_queue.cc:260] T f7e41ddb74d04b06b5e706ad44c1482e P 28dabb1e621d4aedbd18cb5ff0091cea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28dabb1e621d4aedbd18cb5ff0091cea" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44665 } } peers { permanent_uuid: "4fd367b2b2444451b116d2d69f690b77" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35079 } } peers { permanent_uuid: "3892f3eb6ebb436d8bcd0f31647a4c6a" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36553 } }
I20260810 23:26:06.688212 26835 tablet_bootstrap.cc:654] T f7e41ddb74d04b06b5e706ad44c1482e P 4fd367b2b2444451b116d2d69f690b77: Neither blocks nor log segments found. Creating new log.
I20260810 23:26:06.690881 26838 tablet_bootstrap.cc:492] T f7e41ddb74d04b06b5e706ad44c1482e P 3892f3eb6ebb436d8bcd0f31647a4c6a: Bootstrap starting.
I20260810 23:26:06.691433 26835 tablet_bootstrap.cc:492] T f7e41ddb74d04b06b5e706ad44c1482e P 4fd367b2b2444451b116d2d69f690b77: No bootstrap required, opened a new log
I20260810 23:26:06.691604 26835 ts_tablet_manager.cc:1397] T f7e41ddb74d04b06b5e706ad44c1482e P 4fd367b2b2444451b116d2d69f690b77: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:26:06.692399 26835 raft_consensus.cc:348] T f7e41ddb74d04b06b5e706ad44c1482e P 4fd367b2b2444451b116d2d69f690b77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28dabb1e621d4aedbd18cb5ff0091cea" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44665 } } peers { permanent_uuid: "4fd367b2b2444451b116d2d69f690b77" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35079 } } peers { permanent_uuid: "3892f3eb6ebb436d8bcd0f31647a4c6a" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36553 } }
I20260810 23:26:06.692613 26835 raft_consensus.cc:374] T f7e41ddb74d04b06b5e706ad44c1482e P 4fd367b2b2444451b116d2d69f690b77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:26:06.692710 26835 raft_consensus.cc:729] T f7e41ddb74d04b06b5e706ad44c1482e P 4fd367b2b2444451b116d2d69f690b77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd367b2b2444451b116d2d69f690b77, State: Initialized, Role: FOLLOWER
I20260810 23:26:06.692950 26835 consensus_queue.cc:260] T f7e41ddb74d04b06b5e706ad44c1482e P 4fd367b2b2444451b116d2d69f690b77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28dabb1e621d4aedbd18cb5ff0091cea" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44665 } } peers { permanent_uuid: "4fd367b2b2444451b116d2d69f690b77" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35079 } } peers { permanent_uuid: "3892f3eb6ebb436d8bcd0f31647a4c6a" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36553 } }
I20260810 23:26:06.693058 26838 tablet_bootstrap.cc:654] T f7e41ddb74d04b06b5e706ad44c1482e P 3892f3eb6ebb436d8bcd0f31647a4c6a: Neither blocks nor log segments found. Creating new log.
I20260810 23:26:06.693799 26835 ts_tablet_manager.cc:1428] T f7e41ddb74d04b06b5e706ad44c1482e P 4fd367b2b2444451b116d2d69f690b77: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 23:26:06.695245 26833 ts_tablet_manager.cc:1428] T f7e41ddb74d04b06b5e706ad44c1482e P 28dabb1e621d4aedbd18cb5ff0091cea: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260810 23:26:06.698336 26648 heartbeater.cc:503] Master 127.14.175.125:32771 was elected leader, sending a full tablet report...
I20260810 23:26:06.701463 26174 heartbeater.cc:503] Master 127.14.175.125:32771 was elected leader, sending a full tablet report...
I20260810 23:26:06.703711 26845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:06.694390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.14.175.65:46631)
0810 23:26:06.703249 (+ 8859us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:06.703438 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8711,"threads_started":1}
I20260810 23:26:06.702432 26846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:06.695443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.14.175.67:35351)
0810 23:26:06.701955 (+ 6512us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:06.702133 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6350,"thread_start_us":6096,"threads_started":1}
I20260810 23:26:06.705974 26845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:06.705476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.14.175.65:55307)
0810 23:26:06.705654 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:06.705758 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 23:26:06.706229 26846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:06.705262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.14.175.67:60843)
0810 23:26:06.705621 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:06.705718 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246}
I20260810 23:26:06.706677 26838 tablet_bootstrap.cc:492] T f7e41ddb74d04b06b5e706ad44c1482e P 3892f3eb6ebb436d8bcd0f31647a4c6a: No bootstrap required, opened a new log
I20260810 23:26:06.706863 26838 ts_tablet_manager.cc:1397] T f7e41ddb74d04b06b5e706ad44c1482e P 3892f3eb6ebb436d8bcd0f31647a4c6a: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.005s
I20260810 23:26:06.707696 26838 raft_consensus.cc:348] T f7e41ddb74d04b06b5e706ad44c1482e P 3892f3eb6ebb436d8bcd0f31647a4c6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28dabb1e621d4aedbd18cb5ff0091cea" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44665 } } peers { permanent_uuid: "4fd367b2b2444451b116d2d69f690b77" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35079 } } peers { permanent_uuid: "3892f3eb6ebb436d8bcd0f31647a4c6a" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36553 } }
I20260810 23:26:06.707897 26838 raft_consensus.cc:374] T f7e41ddb74d04b06b5e706ad44c1482e P 3892f3eb6ebb436d8bcd0f31647a4c6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:26:06.707970 26838 raft_consensus.cc:729] T f7e41ddb74d04b06b5e706ad44c1482e P 3892f3eb6ebb436d8bcd0f31647a4c6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3892f3eb6ebb436d8bcd0f31647a4c6a, State: Initialized, Role: FOLLOWER
I20260810 23:26:06.708204 26838 consensus_queue.cc:260] T f7e41ddb74d04b06b5e706ad44c1482e P 3892f3eb6ebb436d8bcd0f31647a4c6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28dabb1e621d4aedbd18cb5ff0091cea" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44665 } } peers { permanent_uuid: "4fd367b2b2444451b116d2d69f690b77" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35079 } } peers { permanent_uuid: "3892f3eb6ebb436d8bcd0f31647a4c6a" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36553 } }
I20260810 23:26:06.708935 26838 ts_tablet_manager.cc:1428] T f7e41ddb74d04b06b5e706ad44c1482e P 3892f3eb6ebb436d8bcd0f31647a4c6a: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260810 23:26:06.710338 26850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:06.709326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.14.175.66:45119)
0810 23:26:06.709951 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:06.710125 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":316,"threads_started":1}
I20260810 23:26:06.711133 26409 heartbeater.cc:503] Master 127.14.175.125:32771 was elected leader, sending a full tablet report...
I20260810 23:26:06.711679 26850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:06.711263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.14.175.66:52349)
0810 23:26:06.711416 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:06.711520 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 23:26:06.884892 26841 raft_consensus.cc:482] T f7e41ddb74d04b06b5e706ad44c1482e P 4fd367b2b2444451b116d2d69f690b77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:26:06.885178 26841 raft_consensus.cc:504] T f7e41ddb74d04b06b5e706ad44c1482e P 4fd367b2b2444451b116d2d69f690b77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28dabb1e621d4aedbd18cb5ff0091cea" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44665 } } peers { permanent_uuid: "4fd367b2b2444451b116d2d69f690b77" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35079 } } peers { permanent_uuid: "3892f3eb6ebb436d8bcd0f31647a4c6a" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36553 } }
I20260810 23:26:06.885988 26841 leader_election.cc:288] T f7e41ddb74d04b06b5e706ad44c1482e P 4fd367b2b2444451b116d2d69f690b77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28dabb1e621d4aedbd18cb5ff0091cea (127.14.175.67:44665), 3892f3eb6ebb436d8bcd0f31647a4c6a (127.14.175.66:36553)
I20260810 23:26:06.886590 26327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7e41ddb74d04b06b5e706ad44c1482e" candidate_uuid: "4fd367b2b2444451b116d2d69f690b77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3892f3eb6ebb436d8bcd0f31647a4c6a" is_pre_election: true
I20260810 23:26:06.886641 26577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7e41ddb74d04b06b5e706ad44c1482e" candidate_uuid: "4fd367b2b2444451b116d2d69f690b77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28dabb1e621d4aedbd18cb5ff0091cea" is_pre_election: true
I20260810 23:26:06.886832 26327 raft_consensus.cc:2454] T f7e41ddb74d04b06b5e706ad44c1482e P 3892f3eb6ebb436d8bcd0f31647a4c6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd367b2b2444451b116d2d69f690b77 in term 0.
I20260810 23:26:06.886880 26577 raft_consensus.cc:2454] T f7e41ddb74d04b06b5e706ad44c1482e P 28dabb1e621d4aedbd18cb5ff0091cea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd367b2b2444451b116d2d69f690b77 in term 0.
I20260810 23:26:06.887318 25983 leader_election.cc:302] T f7e41ddb74d04b06b5e706ad44c1482e P 4fd367b2b2444451b116d2d69f690b77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3892f3eb6ebb436d8bcd0f31647a4c6a, 4fd367b2b2444451b116d2d69f690b77; no voters:
I20260810 23:26:06.887596 26841 raft_consensus.cc:2781] T f7e41ddb74d04b06b5e706ad44c1482e P 4fd367b2b2444451b116d2d69f690b77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:26:06.887741 26841 raft_consensus.cc:482] T f7e41ddb74d04b06b5e706ad44c1482e P 4fd367b2b2444451b116d2d69f690b77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:26:06.887840 26841 raft_consensus.cc:3037] T f7e41ddb74d04b06b5e706ad44c1482e P 4fd367b2b2444451b116d2d69f690b77 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:26:06.890064 26841 raft_consensus.cc:504] T f7e41ddb74d04b06b5e706ad44c1482e P 4fd367b2b2444451b116d2d69f690b77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28dabb1e621d4aedbd18cb5ff0091cea" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44665 } } peers { permanent_uuid: "4fd367b2b2444451b116d2d69f690b77" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35079 } } peers { permanent_uuid: "3892f3eb6ebb436d8bcd0f31647a4c6a" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36553 } }
I20260810 23:26:06.890618 26841 leader_election.cc:288] T f7e41ddb74d04b06b5e706ad44c1482e P 4fd367b2b2444451b116d2d69f690b77 [CANDIDATE]: Term 1 election: Requested vote from peers 28dabb1e621d4aedbd18cb5ff0091cea (127.14.175.67:44665), 3892f3eb6ebb436d8bcd0f31647a4c6a (127.14.175.66:36553)
I20260810 23:26:06.891141 26577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7e41ddb74d04b06b5e706ad44c1482e" candidate_uuid: "4fd367b2b2444451b116d2d69f690b77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28dabb1e621d4aedbd18cb5ff0091cea"
I20260810 23:26:06.891213 26327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7e41ddb74d04b06b5e706ad44c1482e" candidate_uuid: "4fd367b2b2444451b116d2d69f690b77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3892f3eb6ebb436d8bcd0f31647a4c6a"
I20260810 23:26:06.891386 26577 raft_consensus.cc:3037] T f7e41ddb74d04b06b5e706ad44c1482e P 28dabb1e621d4aedbd18cb5ff0091cea [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:26:06.891484 26327 raft_consensus.cc:3037] T f7e41ddb74d04b06b5e706ad44c1482e P 3892f3eb6ebb436d8bcd0f31647a4c6a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:26:06.893734 26327 raft_consensus.cc:2454] T f7e41ddb74d04b06b5e706ad44c1482e P 3892f3eb6ebb436d8bcd0f31647a4c6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd367b2b2444451b116d2d69f690b77 in term 1.
I20260810 23:26:06.894188 25983 leader_election.cc:302] T f7e41ddb74d04b06b5e706ad44c1482e P 4fd367b2b2444451b116d2d69f690b77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3892f3eb6ebb436d8bcd0f31647a4c6a, 4fd367b2b2444451b116d2d69f690b77; no voters:
I20260810 23:26:06.894469 26841 raft_consensus.cc:2781] T f7e41ddb74d04b06b5e706ad44c1482e P 4fd367b2b2444451b116d2d69f690b77 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:26:06.894470 26577 raft_consensus.cc:2454] T f7e41ddb74d04b06b5e706ad44c1482e P 28dabb1e621d4aedbd18cb5ff0091cea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd367b2b2444451b116d2d69f690b77 in term 1.
I20260810 23:26:06.897594 26841 raft_consensus.cc:686] T f7e41ddb74d04b06b5e706ad44c1482e P 4fd367b2b2444451b116d2d69f690b77 [term 1 LEADER]: Becoming Leader. State: Replica: 4fd367b2b2444451b116d2d69f690b77, State: Running, Role: LEADER
I20260810 23:26:06.898136 26841 consensus_queue.cc:237] T f7e41ddb74d04b06b5e706ad44c1482e P 4fd367b2b2444451b116d2d69f690b77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28dabb1e621d4aedbd18cb5ff0091cea" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44665 } } peers { permanent_uuid: "4fd367b2b2444451b116d2d69f690b77" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35079 } } peers { permanent_uuid: "3892f3eb6ebb436d8bcd0f31647a4c6a" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36553 } }
I20260810 23:26:06.898716 26845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:06.898224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.14.175.65:38397)
0810 23:26:06.898413 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:06.898513 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 23:26:06.901077 25546 catalog_manager.cc:5495] T f7e41ddb74d04b06b5e706ad44c1482e P 4fd367b2b2444451b116d2d69f690b77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fd367b2b2444451b116d2d69f690b77 (127.14.175.65). New cstate: current_term: 1 leader_uuid: "4fd367b2b2444451b116d2d69f690b77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28dabb1e621d4aedbd18cb5ff0091cea" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd367b2b2444451b116d2d69f690b77" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3892f3eb6ebb436d8bcd0f31647a4c6a" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36553 } health_report { overall_health: UNKNOWN } } }
I20260810 23:26:06.957243 26828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:06.952543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56946 (local address 127.14.175.65:35079)
0810 23:26:06.952732 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:06.952736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:06.952754 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:26:06.952820 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:06.952824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:06.952888 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:06.952964 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:26:06.953375 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:06.954143 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:06.955551 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:06.956015 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:06.956122 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:06.956578 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:26:06.956692 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:26:06.956766 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:06.956901 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 23:26:06.956972 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:26:06.960601 26577 raft_consensus.cc:1261] T f7e41ddb74d04b06b5e706ad44c1482e P 28dabb1e621d4aedbd18cb5ff0091cea [term 1 FOLLOWER]: Refusing update from remote peer 4fd367b2b2444451b116d2d69f690b77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:26:06.960561 26327 raft_consensus.cc:1261] T f7e41ddb74d04b06b5e706ad44c1482e P 3892f3eb6ebb436d8bcd0f31647a4c6a [term 1 FOLLOWER]: Refusing update from remote peer 4fd367b2b2444451b116d2d69f690b77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:26:06.961261 26852 consensus_queue.cc:1035] T f7e41ddb74d04b06b5e706ad44c1482e P 4fd367b2b2444451b116d2d69f690b77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28dabb1e621d4aedbd18cb5ff0091cea" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:26:06.961634 26841 consensus_queue.cc:1035] T f7e41ddb74d04b06b5e706ad44c1482e P 4fd367b2b2444451b116d2d69f690b77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3892f3eb6ebb436d8bcd0f31647a4c6a" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:26:06.964381 26846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:06.963789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.14.175.67:50763)
0810 23:26:06.964037 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:06.964165 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260810 23:26:06.966030 26850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:06.965564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.14.175.66:35975)
0810 23:26:06.965722 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:06.965845 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 23:26:06.968545 26855 mvcc.cc:205] Tried to move back new op lower bound from 7317112287063818240 to 7317112286817087488. Current Snapshot: MvccSnapshot[applied={T|T < 7317112287063818240}]
I20260810 23:26:06.971814 26850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:06.970624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.14.175.66:59653)
0810 23:26:06.971520 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:06.971633 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":766}
I20260810 23:26:06.972416 26857 mvcc.cc:205] Tried to move back new op lower bound from 7317112287063818240 to 7317112286817087488. Current Snapshot: MvccSnapshot[applied={T|T < 7317112287063818240}]
I20260810 23:26:06.984521 26845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:06.983999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.14.175.65:50501)
0810 23:26:06.984183 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:06.984291 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 23:26:06.987026 25545 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:26:06.987350 25545 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:26:06.988018 26845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:06.987477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.14.175.65:53581)
0810 23:26:06.987702 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:06.987804 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260810 23:26:06.991552 26846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:06.991030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.14.175.67:46647)
0810 23:26:06.991264 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:06.991374 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260810 23:26:06.995332 25545 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8e7587e7a104723916c49843b278e0f: Sending DeleteTablet for 3 replicas of tablet f7e41ddb74d04b06b5e706ad44c1482e
I20260810 23:26:06.996733 26526 tablet_service.cc:1514] Processing DeleteTablet for tablet f7e41ddb74d04b06b5e706ad44c1482e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:26:06 UTC) from {username='oryx', principal='oryx/127.14.175.125@KRBTEST.COM'} at 127.0.0.1:38102
I20260810 23:26:06.996951 26064 tablet_service.cc:1514] Processing DeleteTablet for tablet f7e41ddb74d04b06b5e706ad44c1482e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:26:06 UTC) from {username='oryx', principal='oryx/127.14.175.125@KRBTEST.COM'} at 127.0.0.1:56930
I20260810 23:26:06.997257 26296 tablet_service.cc:1514] Processing DeleteTablet for tablet f7e41ddb74d04b06b5e706ad44c1482e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:26:06 UTC) from {username='oryx', principal='oryx/127.14.175.125@KRBTEST.COM'} at 127.0.0.1:32936
I20260810 23:26:06.997263 26524 tablet_service.cc:1514] Processing DeleteTablet for tablet f7e41ddb74d04b06b5e706ad44c1482e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:26:06 UTC) from {username='oryx', principal='oryx/127.14.175.125@KRBTEST.COM'} at 127.0.0.1:38102
I20260810 23:26:06.997471 26062 tablet_service.cc:1514] Processing DeleteTablet for tablet f7e41ddb74d04b06b5e706ad44c1482e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:26:06 UTC) from {username='oryx', principal='oryx/127.14.175.125@KRBTEST.COM'} at 127.0.0.1:56930
I20260810 23:26:06.998416 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 25832
I20260810 23:26:07.001683 26876 tablet_replica.cc:330] T f7e41ddb74d04b06b5e706ad44c1482e P 4fd367b2b2444451b116d2d69f690b77: stopping tablet replica
I20260810 23:26:07.001878 26876 raft_consensus.cc:2229] T f7e41ddb74d04b06b5e706ad44c1482e P 4fd367b2b2444451b116d2d69f690b77 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:26:07.002247 26876 raft_consensus.cc:2258] T f7e41ddb74d04b06b5e706ad44c1482e P 4fd367b2b2444451b116d2d69f690b77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:26:07.002853 26874 tablet_replica.cc:330] T f7e41ddb74d04b06b5e706ad44c1482e P 28dabb1e621d4aedbd18cb5ff0091cea: stopping tablet replica
I20260810 23:26:07.003048 26874 raft_consensus.cc:2229] T f7e41ddb74d04b06b5e706ad44c1482e P 28dabb1e621d4aedbd18cb5ff0091cea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:26:07.003109 26877 tablet_replica.cc:330] T f7e41ddb74d04b06b5e706ad44c1482e P 3892f3eb6ebb436d8bcd0f31647a4c6a: stopping tablet replica
I20260810 23:26:07.003327 26877 raft_consensus.cc:2229] T f7e41ddb74d04b06b5e706ad44c1482e P 3892f3eb6ebb436d8bcd0f31647a4c6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:26:07.003357 26874 raft_consensus.cc:2258] T f7e41ddb74d04b06b5e706ad44c1482e P 28dabb1e621d4aedbd18cb5ff0091cea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:26:07.003648 26877 raft_consensus.cc:2258] T f7e41ddb74d04b06b5e706ad44c1482e P 3892f3eb6ebb436d8bcd0f31647a4c6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:26:07.004024 26876 ts_tablet_manager.cc:1905] T f7e41ddb74d04b06b5e706ad44c1482e P 4fd367b2b2444451b116d2d69f690b77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:26:07.004372 26874 ts_tablet_manager.cc:1905] T f7e41ddb74d04b06b5e706ad44c1482e P 28dabb1e621d4aedbd18cb5ff0091cea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:26:07.005141 26877 ts_tablet_manager.cc:1905] T f7e41ddb74d04b06b5e706ad44c1482e P 3892f3eb6ebb436d8bcd0f31647a4c6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:26:07.010995 26876 ts_tablet_manager.cc:1918] T f7e41ddb74d04b06b5e706ad44c1482e P 4fd367b2b2444451b116d2d69f690b77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:26:07.010995 26874 ts_tablet_manager.cc:1918] T f7e41ddb74d04b06b5e706ad44c1482e P 28dabb1e621d4aedbd18cb5ff0091cea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:26:07.011185 26874 log.cc:1196] T f7e41ddb74d04b06b5e706ad44c1482e P 28dabb1e621d4aedbd18cb5ff0091cea: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-2/wal/wals/f7e41ddb74d04b06b5e706ad44c1482e
I20260810 23:26:07.011185 26876 log.cc:1196] T f7e41ddb74d04b06b5e706ad44c1482e P 4fd367b2b2444451b116d2d69f690b77: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-0/wal/wals/f7e41ddb74d04b06b5e706ad44c1482e
I20260810 23:26:07.011641 26874 ts_tablet_manager.cc:1939] T f7e41ddb74d04b06b5e706ad44c1482e P 28dabb1e621d4aedbd18cb5ff0091cea: Deleting consensus metadata
I20260810 23:26:07.011633 26876 ts_tablet_manager.cc:1939] T f7e41ddb74d04b06b5e706ad44c1482e P 4fd367b2b2444451b116d2d69f690b77: Deleting consensus metadata
I20260810 23:26:07.013005 25522 catalog_manager.cc:4841] TS 4fd367b2b2444451b116d2d69f690b77 (127.14.175.65:35079): tablet f7e41ddb74d04b06b5e706ad44c1482e (table test-table [id=f6654b18c03647aaadd8a3b73c42c3cc]) successfully deleted
I20260810 23:26:07.013350 25523 catalog_manager.cc:4841] TS 28dabb1e621d4aedbd18cb5ff0091cea (127.14.175.67:44665): tablet f7e41ddb74d04b06b5e706ad44c1482e (table test-table [id=f6654b18c03647aaadd8a3b73c42c3cc]) successfully deleted
I20260810 23:26:07.013501 26877 ts_tablet_manager.cc:1918] T f7e41ddb74d04b06b5e706ad44c1482e P 3892f3eb6ebb436d8bcd0f31647a4c6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:26:07.013672 26877 log.cc:1196] T f7e41ddb74d04b06b5e706ad44c1482e P 3892f3eb6ebb436d8bcd0f31647a4c6a: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404299514076-15037-0/minicluster-data/ts-1/wal/wals/f7e41ddb74d04b06b5e706ad44c1482e
W20260810 23:26:07.013756 25523 catalog_manager.cc:4808] TS 28dabb1e621d4aedbd18cb5ff0091cea (127.14.175.67:44665): delete failed for tablet f7e41ddb74d04b06b5e706ad44c1482e because the tablet was not found. No further retry: Not found: Tablet not found: f7e41ddb74d04b06b5e706ad44c1482e
I20260810 23:26:07.014181 26877 ts_tablet_manager.cc:1939] T f7e41ddb74d04b06b5e706ad44c1482e P 3892f3eb6ebb436d8bcd0f31647a4c6a: Deleting consensus metadata
W20260810 23:26:07.014580 25522 catalog_manager.cc:4808] TS 4fd367b2b2444451b116d2d69f690b77 (127.14.175.65:35079): delete failed for tablet f7e41ddb74d04b06b5e706ad44c1482e because the tablet was not found. No further retry: Not found: Tablet not found: f7e41ddb74d04b06b5e706ad44c1482e
I20260810 23:26:07.015368 25526 catalog_manager.cc:4841] TS 3892f3eb6ebb436d8bcd0f31647a4c6a (127.14.175.66:36553): tablet f7e41ddb74d04b06b5e706ad44c1482e (table test-table [id=f6654b18c03647aaadd8a3b73c42c3cc]) successfully deleted
I20260810 23:26:07.026710 26828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:07.000678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56956 (local address 127.14.175.65:35079)
0810 23:26:07.000981 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:07.000988 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:07.012812 (+ 11824us) server_negotiation.cc:407] Connection header received
0810 23:26:07.013361 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:07.013366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:07.013542 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:07.013631 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:07.015760 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:07.016522 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:07.017893 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:07.018308 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:07.018445 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:07.021100 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:07.021117 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:07.021136 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:07.021213 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:07.024046 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:07.024778 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:07.024785 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:07.024787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:07.024927 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:07.025341 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:07.025345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:07.025347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:07.025737 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:07.025970 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:07.026264 (+ 294us) server_negotiation.cc:299] Negotiation successful
0810 23:26:07.026438 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260810 23:26:07.027382 26165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:26:07.150462 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 25832
I20260810 23:26:07.183471 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 26188
I20260810 23:26:07.199530 26881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:07.184216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32938 (local address 127.14.175.66:36553)
0810 23:26:07.184718 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:07.184723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:07.184740 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:26:07.184798 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:07.184802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:07.184861 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:07.184964 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:07.186440 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:07.187212 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:07.189271 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:07.189721 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:07.189842 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:07.193099 (+ 3257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:07.193128 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:07.193203 (+ 75us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:07.193252 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:07.196290 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:07.197247 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:07.197259 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:07.197262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:07.197452 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:07.197949 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:07.197952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:07.197954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:07.198379 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:07.198661 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:07.198956 (+ 295us) server_negotiation.cc:299] Negotiation successful
0810 23:26:07.199151 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":287,"threads_started":1}
I20260810 23:26:07.200280 26401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:26:07.249084 26883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:07.247431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.0.0.1:37606)
0810 23:26:07.248618 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:07.248826 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1036,"thread_start_us":797,"threads_started":1}
I20260810 23:26:07.250479 26883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:07.250068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46435 (local address 127.0.0.1:37610)
0810 23:26:07.250213 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:07.250284 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:46435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 23:26:07.296655 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 26188
I20260810 23:26:07.319608 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 26421
I20260810 23:26:07.337435 26889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:07.320567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38114 (local address 127.14.175.67:44665)
0810 23:26:07.321076 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:07.321081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:07.321100 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:26:07.321182 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:07.321187 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:07.321272 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:07.321368 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:07.322891 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:07.323712 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:07.325459 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:07.326005 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:07.326124 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:07.329110 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:07.329132 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:07.329182 (+ 50us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:07.329239 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:07.334006 (+ 4767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:07.334958 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:07.334969 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:07.334971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:07.335170 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:07.335697 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:07.335702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:07.335704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:07.336177 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:07.336445 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:07.336832 (+ 387us) server_negotiation.cc:299] Negotiation successful
0810 23:26:07.337042 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":270,"threads_started":1}
I20260810 23:26:07.338241 26633 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:26:07.433485 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 26421
I20260810 23:26:07.458477 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 25316
I20260810 23:26:07.459792 25602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:26:07.588248 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 25316
W20260810 23:26:07.619834 25858 connection.cc:535] server connection from 127.0.0.1:36306 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 23:26:07.620163 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 25489
I20260810 23:26:07.621389 25926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:26:07.739310 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 25489
2026-08-10T23:26:07Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (12469 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:26:07 dist-test-slave-rmhg krb5kdc[26900](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:26:07 dist-test-slave-rmhg krb5kdc[26900](info): set up 2 sockets
Aug 10 23:26:07 dist-test-slave-rmhg krb5kdc[26900](info): commencing operation
krb5kdc: starting...
W20260810 23:26:09.000834 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.205s user 0.001s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:26:09 dist-test-slave-rmhg krb5kdc[26900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:26:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:26:09Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.126.keytab.
Entry for principal oryx/127.14.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.126.keytab.
I20260810 23:26:09.272205 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:41621
--webserver_interface=127.14.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.126.keytab
--principal=oryx/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:37147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:41621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:26:09.626977 26919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:26:09.627308 26919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:26:09.627399 26919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:26:09.634690 26919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:26:09.634807 26919 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 23:26:09.634848 26919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:26:09.634908 26919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:26:09.634958 26919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:26:09.635020 26919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:26:09.645210 26919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:37147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:41621
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.126.keytab
--principal=oryx/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:41621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:26:09.646596 26919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:26:09.648000 26919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:26:09.660346 26931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:09.660384 26926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:09.660346 26925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:26:09.661513 26919 server_base.cc:1034] running on GCE node
I20260810 23:26:09.662395 26919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:26:09.663959 26919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:26:09.665253 26919 hybrid_clock.cc:648] HybridClock initialized: now 1786404369665181 us; error 39 us; skew 500 ppm
I20260810 23:26:09.665608 26919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:26:09 dist-test-slave-rmhg krb5kdc[26900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404369, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:26:09.672412 26919 init.cc:377] Logged in from keytab as oryx/127.14.175.126@KRBTEST.COM (short username oryx)
I20260810 23:26:09.674278 26919 webserver.cc:460] Webserver started at http://127.14.175.126:34241/ using document root <none> and password file <none>
I20260810 23:26:09.675045 26919 fs_manager.cc:359] Metadata directory not provided
I20260810 23:26:09.675158 26919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:26:09.675486 26919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:26:09.678673 26919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "b274abd910bd4f9d88d4d17b7bf70fdc"
format_stamp: "Formatted at 2026-08-10 23:26:09 on dist-test-slave-rmhg"
server_key: "1f66ad289b82035e0979d489d97705f9"
server_key_iv: "62797eade2f8ea62e27430452358e63f"
server_key_version: "encryptionkey@0"
I20260810 23:26:09.679529 26919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "b274abd910bd4f9d88d4d17b7bf70fdc"
format_stamp: "Formatted at 2026-08-10 23:26:09 on dist-test-slave-rmhg"
server_key: "1f66ad289b82035e0979d489d97705f9"
server_key_iv: "62797eade2f8ea62e27430452358e63f"
server_key_version: "encryptionkey@0"
I20260810 23:26:09.685002 26919 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 23:26:09.688736 26946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:09.690145 26919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:26:09.690323 26919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "b274abd910bd4f9d88d4d17b7bf70fdc"
format_stamp: "Formatted at 2026-08-10 23:26:09 on dist-test-slave-rmhg"
server_key: "1f66ad289b82035e0979d489d97705f9"
server_key_iv: "62797eade2f8ea62e27430452358e63f"
server_key_version: "encryptionkey@0"
I20260810 23:26:09.690587 26919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:26:09.712559 26919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:26:09.716816 26919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:26:09.717109 26919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:26:09.738880 26919 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:41621
I20260810 23:26:09.738888 27048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:41621 every 8 connection(s)
I20260810 23:26:09.740413 26919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:26:09.744588 27049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:26:09.748704 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 26919
I20260810 23:26:09.748968 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:26:09.749835 15037 external_mini_cluster.cc:1428] Setting key 354c8702b1a829742353fea3f35d2fd3
I20260810 23:26:09.759049 27049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b274abd910bd4f9d88d4d17b7bf70fdc: Bootstrap starting.
I20260810 23:26:09.763309 27049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b274abd910bd4f9d88d4d17b7bf70fdc: Neither blocks nor log segments found. Creating new log.
I20260810 23:26:09.764812 27049 log.cc:824] T 00000000000000000000000000000000 P b274abd910bd4f9d88d4d17b7bf70fdc: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:26:09 dist-test-slave-rmhg krb5kdc[26900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM
I20260810 23:26:09.768348 27049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b274abd910bd4f9d88d4d17b7bf70fdc: No bootstrap required, opened a new log
I20260810 23:26:09.775090 27054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:09.752104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40220 (local address 127.14.175.126:41621)
0810 23:26:09.752911 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:09.752925 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:09.752972 (+ 47us) server_negotiation.cc:407] Connection header received
0810 23:26:09.754742 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:09.754775 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:09.755248 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:09.755623 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:09.757561 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:09.758973 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:09.760760 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:09.761636 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:09.761798 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:09.766892 (+ 5094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:09.766901 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:09.766920 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:09.766965 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:09.770331 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:09.771292 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:09.771300 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:09.771308 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:09.771533 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:09.771986 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:09.771991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:09.771993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:09.772645 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:09.772888 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:09.773453 (+ 565us) server_negotiation.cc:299] Negotiation successful
0810 23:26:09.773923 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":348,"threads_started":1}
I20260810 23:26:09.777853 27049 raft_consensus.cc:348] T 00000000000000000000000000000000 P b274abd910bd4f9d88d4d17b7bf70fdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b274abd910bd4f9d88d4d17b7bf70fdc" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 41621 } }
I20260810 23:26:09.778278 27049 raft_consensus.cc:374] T 00000000000000000000000000000000 P b274abd910bd4f9d88d4d17b7bf70fdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:26:09.778401 27049 raft_consensus.cc:729] T 00000000000000000000000000000000 P b274abd910bd4f9d88d4d17b7bf70fdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b274abd910bd4f9d88d4d17b7bf70fdc, State: Initialized, Role: FOLLOWER
I20260810 23:26:09.779006 27049 consensus_queue.cc:260] T 00000000000000000000000000000000 P b274abd910bd4f9d88d4d17b7bf70fdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b274abd910bd4f9d88d4d17b7bf70fdc" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 41621 } }
I20260810 23:26:09.779242 27049 raft_consensus.cc:388] T 00000000000000000000000000000000 P b274abd910bd4f9d88d4d17b7bf70fdc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:26:09.779377 27049 raft_consensus.cc:482] T 00000000000000000000000000000000 P b274abd910bd4f9d88d4d17b7bf70fdc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:26:09.779536 27049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b274abd910bd4f9d88d4d17b7bf70fdc [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:26:09.781880 27049 raft_consensus.cc:504] T 00000000000000000000000000000000 P b274abd910bd4f9d88d4d17b7bf70fdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b274abd910bd4f9d88d4d17b7bf70fdc" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 41621 } }
I20260810 23:26:09.782473 27049 leader_election.cc:302] T 00000000000000000000000000000000 P b274abd910bd4f9d88d4d17b7bf70fdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b274abd910bd4f9d88d4d17b7bf70fdc; no voters:
I20260810 23:26:09.783151 27049 leader_election.cc:288] T 00000000000000000000000000000000 P b274abd910bd4f9d88d4d17b7bf70fdc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:26:09.783366 27058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b274abd910bd4f9d88d4d17b7bf70fdc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:26:09.784408 27058 raft_consensus.cc:686] T 00000000000000000000000000000000 P b274abd910bd4f9d88d4d17b7bf70fdc [term 1 LEADER]: Becoming Leader. State: Replica: b274abd910bd4f9d88d4d17b7bf70fdc, State: Running, Role: LEADER
I20260810 23:26:09.785194 27058 consensus_queue.cc:237] T 00000000000000000000000000000000 P b274abd910bd4f9d88d4d17b7bf70fdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b274abd910bd4f9d88d4d17b7bf70fdc" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 41621 } }
I20260810 23:26:09.785852 27049 sys_catalog.cc:564] T 00000000000000000000000000000000 P b274abd910bd4f9d88d4d17b7bf70fdc [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:26:09.788497 27059 sys_catalog.cc:455] T 00000000000000000000000000000000 P b274abd910bd4f9d88d4d17b7bf70fdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b274abd910bd4f9d88d4d17b7bf70fdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b274abd910bd4f9d88d4d17b7bf70fdc" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 41621 } } }
I20260810 23:26:09.788336 27061 sys_catalog.cc:455] T 00000000000000000000000000000000 P b274abd910bd4f9d88d4d17b7bf70fdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b274abd910bd4f9d88d4d17b7bf70fdc. Latest consensus state: current_term: 1 leader_uuid: "b274abd910bd4f9d88d4d17b7bf70fdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b274abd910bd4f9d88d4d17b7bf70fdc" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 41621 } } }
I20260810 23:26:09.789142 27059 sys_catalog.cc:458] T 00000000000000000000000000000000 P b274abd910bd4f9d88d4d17b7bf70fdc [sys.catalog]: This master's current role is: LEADER
I20260810 23:26:09.789256 27061 sys_catalog.cc:458] T 00000000000000000000000000000000 P b274abd910bd4f9d88d4d17b7bf70fdc [sys.catalog]: This master's current role is: LEADER
I20260810 23:26:09.792749 27071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:26:09.798177 27071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:26:09.808629 27071 catalog_manager.cc:1347] Generated new cluster ID: 8b5b9f36cc604254b60bc3be27d6cc8c
I20260810 23:26:09.808768 27071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:26:09.818387 27071 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:26:09.819801 27071 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:26:09.832119 27071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b274abd910bd4f9d88d4d17b7bf70fdc: Generated new TSK 0
I20260810 23:26:09.833410 27071 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.65.keytab.
Entry for principal oryx/127.14.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.65.keytab.
I20260810 23:26:09.900918 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.65:0
--local_ip_for_outbound_sockets=127.14.175.65
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:41621
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.65.keytab
--principal=oryx/127.14.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:37147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:26:10.253588 27091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:26:10.253929 27091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:26:10.254061 27091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:26:10.262526 27091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:26:10.262661 27091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:26:10.262794 27091 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:26:10.262900 27091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.65
I20260810 23:26:10.273346 27091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:37147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.65.keytab
--principal=oryx/127.14.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.175.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.175.126:41621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.65
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:26:10.274787 27091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:26:10.276278 27091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:26:10.290314 27101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:10.290542 27099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:10.290583 27107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:26:10.291203 27091 server_base.cc:1034] running on GCE node
I20260810 23:26:10.291980 27091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:26:10.293242 27091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:26:10.294498 27091 hybrid_clock.cc:648] HybridClock initialized: now 1786404370294414 us; error 61 us; skew 500 ppm
I20260810 23:26:10.294871 27091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:26:10 dist-test-slave-rmhg krb5kdc[26900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404370, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:26:10.298375 27091 init.cc:377] Logged in from keytab as oryx/127.14.175.65@KRBTEST.COM (short username oryx)
I20260810 23:26:10.300071 27091 webserver.cc:460] Webserver started at http://127.14.175.65:46715/ using document root <none> and password file <none>
I20260810 23:26:10.300796 27091 fs_manager.cc:359] Metadata directory not provided
I20260810 23:26:10.300889 27091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:26:10.301280 27091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:26:10.304666 27091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-0/data/instance:
uuid: "d45feef75e5b43d0855345b5de047874"
format_stamp: "Formatted at 2026-08-10 23:26:10 on dist-test-slave-rmhg"
server_key: "a7dc25a6b6cdb82a12be24a47948673b"
server_key_iv: "9c3fe23a47b72d73d498136cff4b4b1b"
server_key_version: "encryptionkey@0"
I20260810 23:26:10.305505 27091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance:
uuid: "d45feef75e5b43d0855345b5de047874"
format_stamp: "Formatted at 2026-08-10 23:26:10 on dist-test-slave-rmhg"
server_key: "a7dc25a6b6cdb82a12be24a47948673b"
server_key_iv: "9c3fe23a47b72d73d498136cff4b4b1b"
server_key_version: "encryptionkey@0"
I20260810 23:26:10.311005 27091 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 23:26:10.314867 27117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:10.316306 27091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260810 23:26:10.316457 27091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-0/wal
uuid: "d45feef75e5b43d0855345b5de047874"
format_stamp: "Formatted at 2026-08-10 23:26:10 on dist-test-slave-rmhg"
server_key: "a7dc25a6b6cdb82a12be24a47948673b"
server_key_iv: "9c3fe23a47b72d73d498136cff4b4b1b"
server_key_version: "encryptionkey@0"
I20260810 23:26:10.316669 27091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:26:10.333029 27091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:26:10.337724 27091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:26:10.337977 27091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:26:10.341320 27091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:26:10.341436 27091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:10.341590 27091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:26:10.341671 27091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:26:10 dist-test-slave-rmhg krb5kdc[26900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404370, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM
I20260810 23:26:10.367970 27138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:10.341899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:47647 (local address 127.14.175.126:41621)
0810 23:26:10.342605 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:10.342610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:10.343688 (+ 1078us) server_negotiation.cc:407] Connection header received
0810 23:26:10.345922 (+ 2234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:10.345928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:10.346020 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:10.346121 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:10.348638 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:10.349714 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:10.352866 (+ 3152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:10.353425 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:10.353581 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:10.358822 (+ 5241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:10.358831 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:10.358837 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:10.358891 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:10.362293 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:10.363546 (+ 1253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:10.363552 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:10.363556 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:10.363729 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:10.364405 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:10.364409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:10.364411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:10.364788 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:10.364936 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:10.367497 (+ 2561us) server_negotiation.cc:299] Negotiation successful
0810 23:26:10.367717 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":486,"threads_started":1}
I20260810 23:26:10.368244 27140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:10.342642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:41621 (local address 127.14.175.65:47647)
0810 23:26:10.343506 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:10.343551 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:10.345580 (+ 2029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:10.346449 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:10.346462 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:10.347449 (+ 987us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:10.348422 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:10.348453 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:10.349955 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:10.349965 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:10.352660 (+ 2695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:10.352683 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:10.353749 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:10.353763 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:10.354100 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:10.355015 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:10.355045 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:10.358483 (+ 3438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:10.362527 (+ 4044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:10.362539 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:10.362566 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:10.363380 (+ 814us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:10.363866 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:10.363872 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:10.363876 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:10.364210 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:10.365018 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:10.365025 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:10.365608 (+ 583us) client_negotiation.cc:769] Sending connection context
0810 23:26:10.366025 (+ 417us) client_negotiation.cc:240] Negotiation successful
0810 23:26:10.366709 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":372,"threads_started":1}
I20260810 23:26:10.383879 27091 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.65:46357
I20260810 23:26:10.383992 27322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.65:46357 every 8 connection(s)
I20260810 23:26:10.385594 27091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
I20260810 23:26:10.387917 27323 heartbeater.cc:347] Connected to a master server at 127.14.175.126:41621
I20260810 23:26:10.388198 27323 heartbeater.cc:464] Registering TS with master...
I20260810 23:26:10.388499 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 27091
I20260810 23:26:10.388746 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance
I20260810 23:26:10.389093 27323 heartbeater.cc:511] Master 127.14.175.126:41621 requested a full tablet report, sending...
I20260810 23:26:10.389606 15037 external_mini_cluster.cc:1428] Setting key 8df60f8c9ce7920038940e8e53624d11
I20260810 23:26:10.391692 26980 ts_manager.cc:194] Registered new tserver with Master: d45feef75e5b43d0855345b5de047874 (127.14.175.65:46357)
I20260810 23:26:10.393335 27138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:10.378397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:49699 (local address 127.14.175.126:41621)
0810 23:26:10.378590 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:10.378597 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:10.378867 (+ 270us) server_negotiation.cc:407] Connection header received
0810 23:26:10.379962 (+ 1095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:10.379969 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:10.380077 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:10.380190 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:10.381531 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:10.382354 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:10.384347 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:10.385031 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:10.385205 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:10.387546 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:10.387551 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:10.387555 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:10.387590 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:10.390272 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:10.391060 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:10.391066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:10.391070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:10.391230 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:10.391688 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:10.391693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:10.391696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:10.392069 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:10.392209 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:10.392701 (+ 492us) server_negotiation.cc:299] Negotiation successful
0810 23:26:10.392936 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 23:26:10.393379 27310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:10.377841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:41621 (local address 127.14.175.65:49699)
0810 23:26:10.378700 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:10.378753 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:10.379773 (+ 1020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:10.380247 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:10.380252 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:10.380862 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:10.381317 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:10.381334 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:10.382518 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:10.382522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:10.384173 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:10.384190 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:10.385269 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:10.385274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:10.385418 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:10.386077 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:10.386106 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:10.387365 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:10.390393 (+ 3028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:10.390399 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:10.390404 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:10.390931 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:10.391356 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:10.391361 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:10.391365 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:10.391530 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:10.392201 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:10.392206 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:10.392430 (+ 224us) client_negotiation.cc:769] Sending connection context
0810 23:26:10.392558 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 23:26:10.392899 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":749,"threads_started":1}
I20260810 23:26:10.393827 26980 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.175.65@KRBTEST.COM'} at 127.14.175.65:47647
WARNING: no policy specified for oryx/127.14.175.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.175.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.66.keytab.
Entry for principal oryx/127.14.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.66.keytab.
I20260810 23:26:10.448395 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.66:0
--local_ip_for_outbound_sockets=127.14.175.66
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:41621
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.66.keytab
--principal=oryx/127.14.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:37147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:26:10.796761 27328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:26:10.797072 27328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:26:10.797228 27328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:26:10.804715 27328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:26:10.804840 27328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:26:10.804934 27328 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:26:10.805033 27328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.66
I20260810 23:26:10.815510 27328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:37147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.66.keytab
--principal=oryx/127.14.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.175.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.175.126:41621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.66
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:26:10.816957 27328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:26:10.818511 27328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:26:10.832700 27338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:10.832922 27342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:10.833319 27337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:26:10.833640 27328 server_base.cc:1034] running on GCE node
I20260810 23:26:10.834353 27328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:26:10.835382 27328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:26:10.836617 27328 hybrid_clock.cc:648] HybridClock initialized: now 1786404370836546 us; error 43 us; skew 500 ppm
I20260810 23:26:10.836943 27328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:26:10 dist-test-slave-rmhg krb5kdc[26900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404370, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:26:10.840247 27328 init.cc:377] Logged in from keytab as oryx/127.14.175.66@KRBTEST.COM (short username oryx)
I20260810 23:26:10.841900 27328 webserver.cc:460] Webserver started at http://127.14.175.66:43753/ using document root <none> and password file <none>
I20260810 23:26:10.842607 27328 fs_manager.cc:359] Metadata directory not provided
I20260810 23:26:10.842711 27328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:26:10.843001 27328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:26:10.846103 27328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-1/data/instance:
uuid: "f46de26a8c5f4921a158eca10ad8970e"
format_stamp: "Formatted at 2026-08-10 23:26:10 on dist-test-slave-rmhg"
server_key: "a96a12f32f01108640820405e4404be2"
server_key_iv: "1485390d3431fb173dd924947e35197e"
server_key_version: "encryptionkey@0"
I20260810 23:26:10.846920 27328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance:
uuid: "f46de26a8c5f4921a158eca10ad8970e"
format_stamp: "Formatted at 2026-08-10 23:26:10 on dist-test-slave-rmhg"
server_key: "a96a12f32f01108640820405e4404be2"
server_key_iv: "1485390d3431fb173dd924947e35197e"
server_key_version: "encryptionkey@0"
I20260810 23:26:10.852478 27328 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 23:26:10.856459 27350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:10.858103 27328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:26:10.858297 27328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-1/wal
uuid: "f46de26a8c5f4921a158eca10ad8970e"
format_stamp: "Formatted at 2026-08-10 23:26:10 on dist-test-slave-rmhg"
server_key: "a96a12f32f01108640820405e4404be2"
server_key_iv: "1485390d3431fb173dd924947e35197e"
server_key_version: "encryptionkey@0"
I20260810 23:26:10.858537 27328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:26:10.874065 27328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:26:10.878273 27328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:26:10.878544 27328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:26:10.881441 27328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:26:10.881533 27328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:10.881652 27328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:26:10.881711 27328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:26:10 dist-test-slave-rmhg krb5kdc[26900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404370, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM
I20260810 23:26:10.906816 27138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:10.882153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:48269 (local address 127.14.175.126:41621)
0810 23:26:10.882354 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:10.882360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:10.883958 (+ 1598us) server_negotiation.cc:407] Connection header received
0810 23:26:10.886791 (+ 2833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:10.886799 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:10.886907 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:10.887031 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:10.889536 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:10.890549 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:10.893290 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:10.894050 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:10.894205 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:10.899312 (+ 5107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:10.899318 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:10.899321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:10.899361 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:10.903073 (+ 3712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:10.903896 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:10.903903 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:10.903907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:10.904082 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:10.904514 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:10.904518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:10.904522 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:10.904934 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:10.905113 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:10.906127 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0810 23:26:10.906403 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 23:26:10.907967 27365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:10.882826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:41621 (local address 127.14.175.66:48269)
0810 23:26:10.883743 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:10.883806 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:10.886364 (+ 2558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:10.887313 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:10.887348 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:10.888244 (+ 896us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:10.889241 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:10.889274 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:10.890712 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:10.890718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:10.893092 (+ 2374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:10.893114 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:10.894294 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:10.894302 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:10.894540 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:10.895380 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:10.895425 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:10.898980 (+ 3555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:10.903253 (+ 4273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:10.903265 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:10.903284 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:10.903758 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:10.904183 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:10.904187 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:10.904189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:10.904405 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:10.905092 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:10.905100 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:10.905634 (+ 534us) client_negotiation.cc:769] Sending connection context
0810 23:26:10.905968 (+ 334us) client_negotiation.cc:240] Negotiation successful
0810 23:26:10.906484 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":366,"threads_started":1}
I20260810 23:26:10.929834 27138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:10.914733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:60611 (local address 127.14.175.126:41621)
0810 23:26:10.914895 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:10.914899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:10.915529 (+ 630us) server_negotiation.cc:407] Connection header received
0810 23:26:10.915695 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:10.915701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:10.915793 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:10.915904 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:10.916802 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:10.918033 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:10.919977 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:10.920639 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:10.920792 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:10.923133 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:10.923139 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:10.923142 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:10.923178 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:10.926905 (+ 3727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:10.927618 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:10.927623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:10.927627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:10.927838 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:10.928159 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:10.928165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:10.928168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:10.928678 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:10.928844 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:10.929220 (+ 376us) server_negotiation.cc:299] Negotiation successful
0810 23:26:10.929464 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:26:10.930027 27499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:10.914843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:41621 (local address 127.14.175.66:60611)
0810 23:26:10.915406 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:10.915421 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:10.915557 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:10.915926 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:10.915929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:10.916366 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:10.916669 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:10.916681 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:10.918228 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:10.918233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:10.919800 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:10.919819 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:10.920849 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:10.920855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:10.921001 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:10.921698 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:10.921735 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:10.922897 (+ 1162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:10.927076 (+ 4179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:10.927080 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:10.927083 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:10.927503 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:10.927944 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:10.927947 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:10.927949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:10.928074 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:10.928782 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:10.928785 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:10.928958 (+ 173us) client_negotiation.cc:769] Sending connection context
0810 23:26:10.929068 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 23:26:10.929465 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":299,"threads_started":1}
I20260810 23:26:10.935165 27328 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.66:46177
I20260810 23:26:10.935171 27556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.66:46177 every 8 connection(s)
I20260810 23:26:10.936452 27328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
I20260810 23:26:10.938570 27557 heartbeater.cc:347] Connected to a master server at 127.14.175.126:41621
I20260810 23:26:10.938813 27557 heartbeater.cc:464] Registering TS with master...
I20260810 23:26:10.939476 27557 heartbeater.cc:511] Master 127.14.175.126:41621 requested a full tablet report, sending...
I20260810 23:26:10.940848 26979 ts_manager.cc:194] Registered new tserver with Master: f46de26a8c5f4921a158eca10ad8970e (127.14.175.66:46177)
I20260810 23:26:10.941788 26979 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.175.66@KRBTEST.COM'} at 127.14.175.66:48269
I20260810 23:26:10.946504 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 27328
I20260810 23:26:10.946693 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance
I20260810 23:26:10.947110 15037 external_mini_cluster.cc:1428] Setting key 834038d9052b3aac6aa82e2fce6a61c8
WARNING: no policy specified for oryx/127.14.175.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.175.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.67.keytab.
Entry for principal oryx/127.14.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.67.keytab.
I20260810 23:26:11.004419 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.67:0
--local_ip_for_outbound_sockets=127.14.175.67
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:41621
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.67.keytab
--principal=oryx/127.14.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:37147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:26:11.357646 27563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:26:11.357977 27563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:26:11.358109 27563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:26:11.365688 27563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:26:11.365839 27563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:26:11.365967 27563 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:26:11.366078 27563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.67
I20260810 23:26:11.376260 27563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:37147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.67.keytab
--principal=oryx/127.14.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.175.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.175.126:41621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.67
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:26:11.377676 27563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:26:11.379068 27563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:26:11.393762 27572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:11.393901 27573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:11.393725 27578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:26:11.394691 27563 server_base.cc:1034] running on GCE node
I20260810 23:26:11.395388 27563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:26:11.396384 27563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:26:11.397625 27563 hybrid_clock.cc:648] HybridClock initialized: now 1786404371397550 us; error 45 us; skew 500 ppm
I20260810 23:26:11.397943 27563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:26:11 dist-test-slave-rmhg krb5kdc[26900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404371, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:26:11.401373 27563 init.cc:377] Logged in from keytab as oryx/127.14.175.67@KRBTEST.COM (short username oryx)
I20260810 23:26:11.402789 27323 heartbeater.cc:503] Master 127.14.175.126:41621 was elected leader, sending a full tablet report...
I20260810 23:26:11.403043 27563 webserver.cc:460] Webserver started at http://127.14.175.67:45917/ using document root <none> and password file <none>
I20260810 23:26:11.403887 27563 fs_manager.cc:359] Metadata directory not provided
I20260810 23:26:11.404033 27563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:26:11.404500 27563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:26:11.407713 27563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-2/data/instance:
uuid: "014a92bf31d243febf2877d5869b6138"
format_stamp: "Formatted at 2026-08-10 23:26:11 on dist-test-slave-rmhg"
server_key: "ce0d91fc3ad1edf628589fadee9ed1eb"
server_key_iv: "516df1874dcba7d2a7bdafe0dd327f2d"
server_key_version: "encryptionkey@0"
I20260810 23:26:11.408612 27563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance:
uuid: "014a92bf31d243febf2877d5869b6138"
format_stamp: "Formatted at 2026-08-10 23:26:11 on dist-test-slave-rmhg"
server_key: "ce0d91fc3ad1edf628589fadee9ed1eb"
server_key_iv: "516df1874dcba7d2a7bdafe0dd327f2d"
server_key_version: "encryptionkey@0"
I20260810 23:26:11.414299 27563 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 23:26:11.418119 27589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:11.419699 27563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:26:11.419893 27563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-2/wal
uuid: "014a92bf31d243febf2877d5869b6138"
format_stamp: "Formatted at 2026-08-10 23:26:11 on dist-test-slave-rmhg"
server_key: "ce0d91fc3ad1edf628589fadee9ed1eb"
server_key_iv: "516df1874dcba7d2a7bdafe0dd327f2d"
server_key_version: "encryptionkey@0"
I20260810 23:26:11.420125 27563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:26:11.432354 27563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:26:11.436844 27563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:26:11.437105 27563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:26:11.439824 27563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:26:11.439930 27563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:11.440114 27563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:26:11.440192 27563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:26:11 dist-test-slave-rmhg krb5kdc[26900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404371, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM
I20260810 23:26:11.464567 27604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:11.440720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:41997 (local address 127.14.175.126:41621)
0810 23:26:11.441246 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:11.441251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:11.442430 (+ 1179us) server_negotiation.cc:407] Connection header received
0810 23:26:11.444481 (+ 2051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:11.444487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:11.444577 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:11.444678 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:11.447144 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:11.448231 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:11.450709 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:11.451299 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:11.451430 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:11.456763 (+ 5333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:11.456769 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:11.456773 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:11.456821 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:11.460420 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:11.461663 (+ 1243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:11.461671 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:11.461676 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:11.461889 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:11.462384 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:11.462389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:11.462393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:11.462834 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:11.462999 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:11.463819 (+ 820us) server_negotiation.cc:299] Negotiation successful
0810 23:26:11.464105 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":237,"threads_started":1}
I20260810 23:26:11.465582 27606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:11.441445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:41621 (local address 127.14.175.67:41997)
0810 23:26:11.442260 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:11.442310 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:11.444166 (+ 1856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:11.445002 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:11.445015 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:11.445933 (+ 918us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:11.446941 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:11.446974 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:11.448376 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:11.448385 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:11.450506 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:11.450530 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:11.451600 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:11.451608 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:11.451940 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:11.452822 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:11.452859 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:11.456388 (+ 3529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:11.460638 (+ 4250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:11.460649 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:11.460673 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:11.461474 (+ 801us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:11.462019 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:11.462023 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:11.462025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:11.462273 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:11.462960 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:11.462967 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:11.463393 (+ 426us) client_negotiation.cc:769] Sending connection context
0810 23:26:11.463705 (+ 312us) client_negotiation.cc:240] Negotiation successful
0810 23:26:11.464186 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":333,"threads_started":1}
I20260810 23:26:11.488770 27604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:11.471763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:60091 (local address 127.14.175.126:41621)
0810 23:26:11.471988 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:11.471995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:11.473925 (+ 1930us) server_negotiation.cc:407] Connection header received
0810 23:26:11.474098 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:11.474104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:11.474208 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:11.474317 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:11.475995 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:11.476903 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:11.479199 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:11.479657 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:11.479805 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:11.482313 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:11.482319 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:11.482323 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:11.482358 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:11.485473 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:11.486416 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:11.486422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:11.486425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:11.486596 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:11.487073 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:11.487079 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:11.487082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:11.487426 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:11.487587 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:11.488178 (+ 591us) server_negotiation.cc:299] Negotiation successful
0810 23:26:11.488418 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260810 23:26:11.488963 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:11.471813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:41621 (local address 127.14.175.67:60091)
0810 23:26:11.473811 (+ 1998us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:11.473828 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:11.473954 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:11.474529 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:11.474534 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:11.475297 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:11.475801 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:11.475822 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:11.477070 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:11.477076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:11.479015 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:11.479034 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:11.479902 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:11.479909 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:11.480107 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:11.480792 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:11.480817 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:11.482066 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:11.485617 (+ 3551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:11.485623 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:11.485631 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:11.486275 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:11.486749 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:11.486754 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:11.486757 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:11.486912 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:11.487673 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:11.487682 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:11.487902 (+ 220us) client_negotiation.cc:769] Sending connection context
0810 23:26:11.488046 (+ 144us) client_negotiation.cc:240] Negotiation successful
0810 23:26:11.488429 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1858,"thread_start_us":1610,"threads_started":1}
I20260810 23:26:11.492427 27563 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.67:40495
I20260810 23:26:11.492506 27787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.67:40495 every 8 connection(s)
I20260810 23:26:11.493974 27563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
I20260810 23:26:11.496095 27788 heartbeater.cc:347] Connected to a master server at 127.14.175.126:41621
I20260810 23:26:11.496315 27788 heartbeater.cc:464] Registering TS with master...
I20260810 23:26:11.497004 27788 heartbeater.cc:511] Master 127.14.175.126:41621 requested a full tablet report, sending...
I20260810 23:26:11.498420 26979 ts_manager.cc:194] Registered new tserver with Master: 014a92bf31d243febf2877d5869b6138 (127.14.175.67:40495)
I20260810 23:26:11.499245 26979 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.175.67@KRBTEST.COM'} at 127.14.175.67:41997
I20260810 23:26:11.502978 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 27563
I20260810 23:26:11.503153 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance
I20260810 23:26:11.503509 15037 external_mini_cluster.cc:1428] Setting key e427bbd610fbc7dc0272b587c4b4fbc1
I20260810 23:26:11.506779 15037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 23:26:11 dist-test-slave-rmhg krb5kdc[26900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404371, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 23:26:11 dist-test-slave-rmhg krb5kdc[26900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:11 dist-test-slave-rmhg krb5kdc[26900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 23:26:11.568449 27604 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:11.557175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40224 (local address 127.14.175.126:41621)
0810 23:26:11.557429 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:11.557436 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:11.557658 (+ 222us) server_negotiation.cc:407] Connection header received
0810 23:26:11.557864 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:11.557868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:11.557940 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:11.558030 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:11.559607 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:11.560406 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:11.561772 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:11.562249 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:11.562373 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:11.568096 (+ 5723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40224: BlockingRecv error: recv got EOF from 127.0.0.1:40224 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 10 23:26:11 dist-test-slave-rmhg krb5kdc[26900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404371, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM
I20260810 23:26:11.587327 27604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:11.572448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40240 (local address 127.14.175.126:41621)
0810 23:26:11.572723 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:11.572730 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:11.572975 (+ 245us) server_negotiation.cc:407] Connection header received
0810 23:26:11.573222 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:11.573226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:11.573295 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:11.573384 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:11.574654 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:11.575408 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:11.576781 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:11.577313 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:11.577445 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:11.581681 (+ 4236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:11.581687 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:11.581690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:11.581738 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:11.584426 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:11.585275 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:11.585282 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:11.585285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:11.585440 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:11.585879 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:11.585884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:11.585886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:11.586197 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:11.586514 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:11.586865 (+ 351us) server_negotiation.cc:299] Negotiation successful
0810 23:26:11.587082 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 23:26:11.591890 26979 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:26:11.599423 26979 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:26:11.636586 27817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:11.612708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:46357 (local address 127.0.0.1:58188)
0810 23:26:11.624535 (+ 11827us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:11.624568 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:11.624775 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:11.629584 (+ 4809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:11.629589 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:11.629621 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:26:11.630037 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:11.630049 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:11.631810 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:11.631815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:11.633905 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:11.633923 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:11.635403 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:11.635410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:11.635656 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:11.635731 (+ 75us) client_negotiation.cc:769] Sending connection context
0810 23:26:11.635880 (+ 149us) client_negotiation.cc:240] Negotiation successful
0810 23:26:11.636093 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11609,"mutex_wait_us":26,"thread_start_us":569,"threads_started":1}
I20260810 23:26:11.637538 27820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:11.613181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58188 (local address 127.14.175.65:46357)
0810 23:26:11.628934 (+ 15753us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:11.628944 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:11.628975 (+ 31us) server_negotiation.cc:407] Connection header received
0810 23:26:11.629095 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:11.629104 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:11.629383 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:11.629601 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:26:11.630237 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:11.631636 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:11.634128 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:11.635172 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:11.635307 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:11.635560 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:11.637005 (+ 1445us) server_negotiation.cc:299] Negotiation successful
0810 23:26:11.637250 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15552,"thread_start_us":16226,"threads_started":1}
I20260810 23:26:11.637542 27822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:11.614116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:46177 (local address 127.0.0.1:41168)
0810 23:26:11.627300 (+ 13184us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:11.627327 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:11.627485 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:11.629067 (+ 1582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:11.629075 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:11.629108 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:26:11.629561 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:11.629576 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:11.631319 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:11.631324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:11.634037 (+ 2713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:11.634053 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:11.636142 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:11.636148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:11.636318 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:11.636353 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 23:26:11.636436 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 23:26:11.636592 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13018,"mutex_wait_us":56,"thread_start_us":13919,"threads_started":1}
I20260810 23:26:11.637979 27826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:11.614414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41168 (local address 127.14.175.66:46177)
0810 23:26:11.628002 (+ 13588us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:11.628012 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:11.628050 (+ 38us) server_negotiation.cc:407] Connection header received
0810 23:26:11.628183 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:11.628192 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:11.628512 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:11.628756 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:26:11.629774 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:11.631175 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:11.634223 (+ 3048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:11.635619 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:11.635746 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:11.635953 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:11.636738 (+ 785us) server_negotiation.cc:299] Negotiation successful
0810 23:26:11.636929 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13434,"thread_start_us":13302,"threads_started":1}
I20260810 23:26:11.638409 27816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:11.612684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:40495 (local address 127.0.0.1:54782)
0810 23:26:11.613659 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:11.613698 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:11.613886 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:11.630721 (+ 16835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:11.630725 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:11.630750 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:26:11.631126 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:11.631139 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:11.633182 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:11.633188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:11.635555 (+ 2367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:11.635576 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:11.637644 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:11.637651 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:11.637818 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:11.637850 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 23:26:11.637931 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 23:26:11.638076 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":61,"thread_start_us":11021,"threads_started":1}
I20260810 23:26:11.638674 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:11.613248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54782 (local address 127.14.175.67:40495)
0810 23:26:11.630186 (+ 16938us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:11.630195 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:11.630229 (+ 34us) server_negotiation.cc:407] Connection header received
0810 23:26:11.630323 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:11.630329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:11.630525 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:11.630717 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:26:11.631334 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:11.632991 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:11.635769 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:11.637403 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:11.637564 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:11.637940 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:11.638016 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 23:26:11.638264 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16748,"thread_start_us":16931,"threads_started":1}
I20260810 23:26:11.641927 27201 tablet_service.cc:1467] Processing CreateTablet for tablet d16193a9ece14808887bb350c2840a18 (DEFAULT_TABLE table=test-table [id=4a1330d780cb417b8d141f5c56a7e49c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:26:11.641878 27436 tablet_service.cc:1467] Processing CreateTablet for tablet d16193a9ece14808887bb350c2840a18 (DEFAULT_TABLE table=test-table [id=4a1330d780cb417b8d141f5c56a7e49c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:26:11.642292 27663 tablet_service.cc:1467] Processing CreateTablet for tablet d16193a9ece14808887bb350c2840a18 (DEFAULT_TABLE table=test-table [id=4a1330d780cb417b8d141f5c56a7e49c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:26:11.643318 27201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d16193a9ece14808887bb350c2840a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:26:11.643376 27436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d16193a9ece14808887bb350c2840a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:26:11.643558 27663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d16193a9ece14808887bb350c2840a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:26:11.655758 27831 tablet_bootstrap.cc:492] T d16193a9ece14808887bb350c2840a18 P d45feef75e5b43d0855345b5de047874: Bootstrap starting.
I20260810 23:26:11.659304 27831 tablet_bootstrap.cc:654] T d16193a9ece14808887bb350c2840a18 P d45feef75e5b43d0855345b5de047874: Neither blocks nor log segments found. Creating new log.
I20260810 23:26:11.660089 27837 tablet_bootstrap.cc:492] T d16193a9ece14808887bb350c2840a18 P 014a92bf31d243febf2877d5869b6138: Bootstrap starting.
I20260810 23:26:11.661772 27831 log.cc:824] T d16193a9ece14808887bb350c2840a18 P d45feef75e5b43d0855345b5de047874: Log is configured to *not* fsync() on all Append() calls
I20260810 23:26:11.661890 27836 tablet_bootstrap.cc:492] T d16193a9ece14808887bb350c2840a18 P f46de26a8c5f4921a158eca10ad8970e: Bootstrap starting.
I20260810 23:26:11.665089 27836 tablet_bootstrap.cc:654] T d16193a9ece14808887bb350c2840a18 P f46de26a8c5f4921a158eca10ad8970e: Neither blocks nor log segments found. Creating new log.
I20260810 23:26:11.665292 27837 tablet_bootstrap.cc:654] T d16193a9ece14808887bb350c2840a18 P 014a92bf31d243febf2877d5869b6138: Neither blocks nor log segments found. Creating new log.
I20260810 23:26:11.666146 27831 tablet_bootstrap.cc:492] T d16193a9ece14808887bb350c2840a18 P d45feef75e5b43d0855345b5de047874: No bootstrap required, opened a new log
I20260810 23:26:11.666215 27836 log.cc:824] T d16193a9ece14808887bb350c2840a18 P f46de26a8c5f4921a158eca10ad8970e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:26:11.666476 27831 ts_tablet_manager.cc:1397] T d16193a9ece14808887bb350c2840a18 P d45feef75e5b43d0855345b5de047874: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260810 23:26:11.666594 27837 log.cc:824] T d16193a9ece14808887bb350c2840a18 P 014a92bf31d243febf2877d5869b6138: Log is configured to *not* fsync() on all Append() calls
I20260810 23:26:11.669385 27836 tablet_bootstrap.cc:492] T d16193a9ece14808887bb350c2840a18 P f46de26a8c5f4921a158eca10ad8970e: No bootstrap required, opened a new log
I20260810 23:26:11.669677 27836 ts_tablet_manager.cc:1397] T d16193a9ece14808887bb350c2840a18 P f46de26a8c5f4921a158eca10ad8970e: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260810 23:26:11.671427 27837 tablet_bootstrap.cc:492] T d16193a9ece14808887bb350c2840a18 P 014a92bf31d243febf2877d5869b6138: No bootstrap required, opened a new log
I20260810 23:26:11.671756 27837 ts_tablet_manager.cc:1397] T d16193a9ece14808887bb350c2840a18 P 014a92bf31d243febf2877d5869b6138: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.007s
I20260810 23:26:11.677266 27836 raft_consensus.cc:348] T d16193a9ece14808887bb350c2840a18 P f46de26a8c5f4921a158eca10ad8970e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } } peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } }
I20260810 23:26:11.677282 27831 raft_consensus.cc:348] T d16193a9ece14808887bb350c2840a18 P d45feef75e5b43d0855345b5de047874 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } } peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } }
I20260810 23:26:11.677670 27836 raft_consensus.cc:374] T d16193a9ece14808887bb350c2840a18 P f46de26a8c5f4921a158eca10ad8970e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:26:11.677742 27831 raft_consensus.cc:374] T d16193a9ece14808887bb350c2840a18 P d45feef75e5b43d0855345b5de047874 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:26:11.677765 27836 raft_consensus.cc:729] T d16193a9ece14808887bb350c2840a18 P f46de26a8c5f4921a158eca10ad8970e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46de26a8c5f4921a158eca10ad8970e, State: Initialized, Role: FOLLOWER
I20260810 23:26:11.677837 27831 raft_consensus.cc:729] T d16193a9ece14808887bb350c2840a18 P d45feef75e5b43d0855345b5de047874 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d45feef75e5b43d0855345b5de047874, State: Initialized, Role: FOLLOWER
I20260810 23:26:11.678388 27836 consensus_queue.cc:260] T d16193a9ece14808887bb350c2840a18 P f46de26a8c5f4921a158eca10ad8970e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } } peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } }
I20260810 23:26:11.678550 27831 consensus_queue.cc:260] T d16193a9ece14808887bb350c2840a18 P d45feef75e5b43d0855345b5de047874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } } peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } }
I20260810 23:26:11.679590 27557 heartbeater.cc:503] Master 127.14.175.126:41621 was elected leader, sending a full tablet report...
I20260810 23:26:11.680313 27836 ts_tablet_manager.cc:1428] T d16193a9ece14808887bb350c2840a18 P f46de26a8c5f4921a158eca10ad8970e: Time spent starting tablet: real 0.011s user 0.009s sys 0.001s
I20260810 23:26:11.680702 27831 ts_tablet_manager.cc:1428] T d16193a9ece14808887bb350c2840a18 P d45feef75e5b43d0855345b5de047874: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260810 23:26:11.683739 27837 raft_consensus.cc:348] T d16193a9ece14808887bb350c2840a18 P 014a92bf31d243febf2877d5869b6138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } } peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } }
I20260810 23:26:11.684116 27837 raft_consensus.cc:374] T d16193a9ece14808887bb350c2840a18 P 014a92bf31d243febf2877d5869b6138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:26:11.684219 27837 raft_consensus.cc:729] T d16193a9ece14808887bb350c2840a18 P 014a92bf31d243febf2877d5869b6138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014a92bf31d243febf2877d5869b6138, State: Initialized, Role: FOLLOWER
I20260810 23:26:11.685567 27837 consensus_queue.cc:260] T d16193a9ece14808887bb350c2840a18 P 014a92bf31d243febf2877d5869b6138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } } peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } }
I20260810 23:26:11.686800 27788 heartbeater.cc:503] Master 127.14.175.126:41621 was elected leader, sending a full tablet report...
I20260810 23:26:11.687532 27837 ts_tablet_manager.cc:1428] T d16193a9ece14808887bb350c2840a18 P 014a92bf31d243febf2877d5869b6138: Time spent starting tablet: real 0.016s user 0.012s sys 0.000s
W20260810 23:26:11.688447 27559 tablet.cc:2369] T d16193a9ece14808887bb350c2840a18 P f46de26a8c5f4921a158eca10ad8970e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:26:11.745702 27789 tablet.cc:2369] T d16193a9ece14808887bb350c2840a18 P 014a92bf31d243febf2877d5869b6138: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:26:11.761991 27844 raft_consensus.cc:482] T d16193a9ece14808887bb350c2840a18 P 014a92bf31d243febf2877d5869b6138 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:26:11.762324 27844 raft_consensus.cc:504] T d16193a9ece14808887bb350c2840a18 P 014a92bf31d243febf2877d5869b6138 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } } peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } }
I20260810 23:26:11.764183 27844 leader_election.cc:288] T d16193a9ece14808887bb350c2840a18 P 014a92bf31d243febf2877d5869b6138 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d45feef75e5b43d0855345b5de047874 (127.14.175.65:46357), f46de26a8c5f4921a158eca10ad8970e (127.14.175.66:46177)
I20260810 23:26:11.771337 27606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:11.764544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:46357 (local address 127.14.175.67:39425)
0810 23:26:11.764739 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:11.764757 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:11.764885 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:11.765527 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:11.765532 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:11.765563 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:26:11.765745 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:11.765754 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:11.767088 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:11.767094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:11.769086 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:11.769103 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:11.770671 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:11.770676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:11.770830 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:11.770858 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 23:26:11.770936 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 23:26:11.771107 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 23:26:11.771385 27820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:11.764755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:39425 (local address 127.14.175.65:46357)
0810 23:26:11.764984 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:11.764990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:11.765010 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:26:11.765087 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:11.765092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:11.765198 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:11.765293 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:26:11.765921 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:11.766963 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:11.769354 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:11.770509 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:11.770683 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:11.770760 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:11.771001 (+ 241us) server_negotiation.cc:299] Negotiation successful
0810 23:26:11.771146 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 23:26:11.772220 27826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:11.765311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:35103 (local address 127.14.175.66:46177)
0810 23:26:11.765531 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:11.765537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:11.765816 (+ 279us) server_negotiation.cc:407] Connection header received
0810 23:26:11.765962 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:11.765966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:11.766034 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:11.766102 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:26:11.766749 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:11.767854 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:11.770078 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:11.771261 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:11.771442 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:11.771529 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:11.771898 (+ 369us) server_negotiation.cc:299] Negotiation successful
0810 23:26:11.772036 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 23:26:11.772303 27846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:11.765003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:46177 (local address 127.14.175.67:35103)
0810 23:26:11.765706 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:11.765724 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:11.765841 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:11.766141 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:11.766146 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:11.766174 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:26:11.766543 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:11.766555 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:11.768015 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:11.768021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:11.769885 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:11.769901 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:11.771483 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:11.771490 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:11.771715 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:11.771750 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 23:26:11.771847 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 23:26:11.771975 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":353,"threads_started":1}
I20260810 23:26:11.772799 27227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16193a9ece14808887bb350c2840a18" candidate_uuid: "014a92bf31d243febf2877d5869b6138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d45feef75e5b43d0855345b5de047874" is_pre_election: true
I20260810 23:26:11.773389 27227 raft_consensus.cc:2454] T d16193a9ece14808887bb350c2840a18 P d45feef75e5b43d0855345b5de047874 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 014a92bf31d243febf2877d5869b6138 in term 0.
I20260810 23:26:11.773402 27475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16193a9ece14808887bb350c2840a18" candidate_uuid: "014a92bf31d243febf2877d5869b6138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46de26a8c5f4921a158eca10ad8970e" is_pre_election: true
I20260810 23:26:11.773881 27475 raft_consensus.cc:2454] T d16193a9ece14808887bb350c2840a18 P f46de26a8c5f4921a158eca10ad8970e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 014a92bf31d243febf2877d5869b6138 in term 0.
I20260810 23:26:11.774138 27591 leader_election.cc:302] T d16193a9ece14808887bb350c2840a18 P 014a92bf31d243febf2877d5869b6138 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 014a92bf31d243febf2877d5869b6138, d45feef75e5b43d0855345b5de047874; no voters:
I20260810 23:26:11.774600 27844 raft_consensus.cc:2781] T d16193a9ece14808887bb350c2840a18 P 014a92bf31d243febf2877d5869b6138 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:26:11.774760 27844 raft_consensus.cc:482] T d16193a9ece14808887bb350c2840a18 P 014a92bf31d243febf2877d5869b6138 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:26:11.774885 27844 raft_consensus.cc:3037] T d16193a9ece14808887bb350c2840a18 P 014a92bf31d243febf2877d5869b6138 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:26:11.777127 27844 raft_consensus.cc:504] T d16193a9ece14808887bb350c2840a18 P 014a92bf31d243febf2877d5869b6138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } } peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } }
I20260810 23:26:11.777781 27844 leader_election.cc:288] T d16193a9ece14808887bb350c2840a18 P 014a92bf31d243febf2877d5869b6138 [CANDIDATE]: Term 1 election: Requested vote from peers d45feef75e5b43d0855345b5de047874 (127.14.175.65:46357), f46de26a8c5f4921a158eca10ad8970e (127.14.175.66:46177)
I20260810 23:26:11.778362 27227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16193a9ece14808887bb350c2840a18" candidate_uuid: "014a92bf31d243febf2877d5869b6138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d45feef75e5b43d0855345b5de047874"
I20260810 23:26:11.778378 27475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16193a9ece14808887bb350c2840a18" candidate_uuid: "014a92bf31d243febf2877d5869b6138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46de26a8c5f4921a158eca10ad8970e"
I20260810 23:26:11.778589 27475 raft_consensus.cc:3037] T d16193a9ece14808887bb350c2840a18 P f46de26a8c5f4921a158eca10ad8970e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:26:11.778566 27227 raft_consensus.cc:3037] T d16193a9ece14808887bb350c2840a18 P d45feef75e5b43d0855345b5de047874 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:26:11.780871 27475 raft_consensus.cc:2454] T d16193a9ece14808887bb350c2840a18 P f46de26a8c5f4921a158eca10ad8970e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 014a92bf31d243febf2877d5869b6138 in term 1.
I20260810 23:26:11.780874 27227 raft_consensus.cc:2454] T d16193a9ece14808887bb350c2840a18 P d45feef75e5b43d0855345b5de047874 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 014a92bf31d243febf2877d5869b6138 in term 1.
I20260810 23:26:11.781342 27591 leader_election.cc:302] T d16193a9ece14808887bb350c2840a18 P 014a92bf31d243febf2877d5869b6138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 014a92bf31d243febf2877d5869b6138, f46de26a8c5f4921a158eca10ad8970e; no voters:
I20260810 23:26:11.781661 27844 raft_consensus.cc:2781] T d16193a9ece14808887bb350c2840a18 P 014a92bf31d243febf2877d5869b6138 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:26:11.782101 27844 raft_consensus.cc:686] T d16193a9ece14808887bb350c2840a18 P 014a92bf31d243febf2877d5869b6138 [term 1 LEADER]: Becoming Leader. State: Replica: 014a92bf31d243febf2877d5869b6138, State: Running, Role: LEADER
I20260810 23:26:11.782693 27844 consensus_queue.cc:237] T d16193a9ece14808887bb350c2840a18 P 014a92bf31d243febf2877d5869b6138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } } peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } }
I20260810 23:26:11.787806 26979 catalog_manager.cc:5495] T d16193a9ece14808887bb350c2840a18 P 014a92bf31d243febf2877d5869b6138 reported cstate change: term changed from 0 to 1, leader changed from <none> to 014a92bf31d243febf2877d5869b6138 (127.14.175.67). New cstate: current_term: 1 leader_uuid: "014a92bf31d243febf2877d5869b6138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } health_report { overall_health: UNKNOWN } } }
I20260810 23:26:11.837750 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:11.832364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54786 (local address 127.14.175.67:40495)
0810 23:26:11.832563 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:11.832568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:11.832588 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:26:11.832718 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:11.832722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:11.832804 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:11.832945 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:26:11.833396 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:11.834132 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:11.835524 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:11.836035 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:11.836131 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:11.836543 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:26:11.837199 (+ 656us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:26:11.837331 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:11.837430 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 23:26:11.837510 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 23:26:11.846952 27475 raft_consensus.cc:1261] T d16193a9ece14808887bb350c2840a18 P f46de26a8c5f4921a158eca10ad8970e [term 1 FOLLOWER]: Refusing update from remote peer 014a92bf31d243febf2877d5869b6138: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:26:11.847142 27227 raft_consensus.cc:1261] T d16193a9ece14808887bb350c2840a18 P d45feef75e5b43d0855345b5de047874 [term 1 FOLLOWER]: Refusing update from remote peer 014a92bf31d243febf2877d5869b6138: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:26:11.848121 27847 consensus_queue.cc:1035] T d16193a9ece14808887bb350c2840a18 P 014a92bf31d243febf2877d5869b6138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:26:11.848507 27844 consensus_queue.cc:1035] T d16193a9ece14808887bb350c2840a18 P 014a92bf31d243febf2877d5869b6138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:26:11.863039 27857 mvcc.cc:205] Tried to move back new op lower bound from 7317112307074064384 to 7317112306826067968. Current Snapshot: MvccSnapshot[applied={T|T < 7317112307074064384}]
I20260810 23:26:11.877089 26979 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:26:11.877408 26979 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:26:11.881358 26979 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b274abd910bd4f9d88d4d17b7bf70fdc: Sending DeleteTablet for 3 replicas of tablet d16193a9ece14808887bb350c2840a18
I20260810 23:26:11.882884 27663 tablet_service.cc:1514] Processing DeleteTablet for tablet d16193a9ece14808887bb350c2840a18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:26:11 UTC) from {username='oryx', principal='oryx/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:54782
I20260810 23:26:11.882876 27201 tablet_service.cc:1514] Processing DeleteTablet for tablet d16193a9ece14808887bb350c2840a18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:26:11 UTC) from {username='oryx', principal='oryx/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:58188
I20260810 23:26:11.883023 27436 tablet_service.cc:1514] Processing DeleteTablet for tablet d16193a9ece14808887bb350c2840a18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:26:11 UTC) from {username='oryx', principal='oryx/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:41168
I20260810 23:26:11.884999 26979 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:26:11.885859 26979 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:26:11.894712 27878 tablet_replica.cc:330] T d16193a9ece14808887bb350c2840a18 P f46de26a8c5f4921a158eca10ad8970e: stopping tablet replica
I20260810 23:26:11.895473 27878 raft_consensus.cc:2229] T d16193a9ece14808887bb350c2840a18 P f46de26a8c5f4921a158eca10ad8970e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:26:11.896234 27878 raft_consensus.cc:2258] T d16193a9ece14808887bb350c2840a18 P f46de26a8c5f4921a158eca10ad8970e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:26:11.898737 27436 tablet_service.cc:1467] Processing CreateTablet for tablet 379dbe6222c0424fba410e06046e625d (DEFAULT_TABLE table=test-table [id=6f6c84df7d6d4f49b79c5cc822386596]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:26:11.898696 27198 tablet_service.cc:1467] Processing CreateTablet for tablet 379dbe6222c0424fba410e06046e625d (DEFAULT_TABLE table=test-table [id=6f6c84df7d6d4f49b79c5cc822386596]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:26:11.898701 27662 tablet_service.cc:1467] Processing CreateTablet for tablet 379dbe6222c0424fba410e06046e625d (DEFAULT_TABLE table=test-table [id=6f6c84df7d6d4f49b79c5cc822386596]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:26:11.899318 27198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 379dbe6222c0424fba410e06046e625d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:26:11.899348 27662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 379dbe6222c0424fba410e06046e625d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:26:11.899505 27436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 379dbe6222c0424fba410e06046e625d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:26:11.906368 27878 ts_tablet_manager.cc:1905] T d16193a9ece14808887bb350c2840a18 P f46de26a8c5f4921a158eca10ad8970e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:26:11.906859 27831 tablet_bootstrap.cc:492] T 379dbe6222c0424fba410e06046e625d P d45feef75e5b43d0855345b5de047874: Bootstrap starting.
I20260810 23:26:11.908682 27831 tablet_bootstrap.cc:654] T 379dbe6222c0424fba410e06046e625d P d45feef75e5b43d0855345b5de047874: Neither blocks nor log segments found. Creating new log.
I20260810 23:26:11.909960 27837 tablet_bootstrap.cc:492] T 379dbe6222c0424fba410e06046e625d P 014a92bf31d243febf2877d5869b6138: Bootstrap starting.
I20260810 23:26:11.911804 27837 tablet_bootstrap.cc:654] T 379dbe6222c0424fba410e06046e625d P 014a92bf31d243febf2877d5869b6138: Neither blocks nor log segments found. Creating new log.
I20260810 23:26:11.912962 27877 tablet_replica.cc:330] T d16193a9ece14808887bb350c2840a18 P 014a92bf31d243febf2877d5869b6138: stopping tablet replica
I20260810 23:26:11.913522 27877 raft_consensus.cc:2229] T d16193a9ece14808887bb350c2840a18 P 014a92bf31d243febf2877d5869b6138 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:26:11.914393 27877 raft_consensus.cc:2258] T d16193a9ece14808887bb350c2840a18 P 014a92bf31d243febf2877d5869b6138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:26:11.917205 27877 ts_tablet_manager.cc:1905] T d16193a9ece14808887bb350c2840a18 P 014a92bf31d243febf2877d5869b6138: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:26:11.917846 27875 tablet_replica.cc:330] T d16193a9ece14808887bb350c2840a18 P d45feef75e5b43d0855345b5de047874: stopping tablet replica
I20260810 23:26:11.918349 27875 raft_consensus.cc:2229] T d16193a9ece14808887bb350c2840a18 P d45feef75e5b43d0855345b5de047874 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:26:11.920300 27875 raft_consensus.cc:2258] T d16193a9ece14808887bb350c2840a18 P d45feef75e5b43d0855345b5de047874 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:26:11.924715 27836 tablet_bootstrap.cc:492] T 379dbe6222c0424fba410e06046e625d P f46de26a8c5f4921a158eca10ad8970e: Bootstrap starting.
I20260810 23:26:11.925133 27877 ts_tablet_manager.cc:1918] T d16193a9ece14808887bb350c2840a18 P 014a92bf31d243febf2877d5869b6138: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:26:11.925348 27877 log.cc:1196] T d16193a9ece14808887bb350c2840a18 P 014a92bf31d243febf2877d5869b6138: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-2/wal/wals/d16193a9ece14808887bb350c2840a18
I20260810 23:26:11.925952 27877 ts_tablet_manager.cc:1939] T d16193a9ece14808887bb350c2840a18 P 014a92bf31d243febf2877d5869b6138: Deleting consensus metadata
I20260810 23:26:11.926590 27836 tablet_bootstrap.cc:654] T 379dbe6222c0424fba410e06046e625d P f46de26a8c5f4921a158eca10ad8970e: Neither blocks nor log segments found. Creating new log.
I20260810 23:26:11.928077 26956 catalog_manager.cc:4841] TS 014a92bf31d243febf2877d5869b6138 (127.14.175.67:40495): tablet d16193a9ece14808887bb350c2840a18 (table test-table [id=4a1330d780cb417b8d141f5c56a7e49c]) successfully deleted
I20260810 23:26:11.930538 27875 ts_tablet_manager.cc:1905] T d16193a9ece14808887bb350c2840a18 P d45feef75e5b43d0855345b5de047874: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:26:11.935731 27878 ts_tablet_manager.cc:1918] T d16193a9ece14808887bb350c2840a18 P f46de26a8c5f4921a158eca10ad8970e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:26:11.935912 27878 log.cc:1196] T d16193a9ece14808887bb350c2840a18 P f46de26a8c5f4921a158eca10ad8970e: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-1/wal/wals/d16193a9ece14808887bb350c2840a18
I20260810 23:26:11.936473 27878 ts_tablet_manager.cc:1939] T d16193a9ece14808887bb350c2840a18 P f46de26a8c5f4921a158eca10ad8970e: Deleting consensus metadata
I20260810 23:26:11.939760 26953 catalog_manager.cc:4841] TS f46de26a8c5f4921a158eca10ad8970e (127.14.175.66:46177): tablet d16193a9ece14808887bb350c2840a18 (table test-table [id=4a1330d780cb417b8d141f5c56a7e49c]) successfully deleted
I20260810 23:26:11.940876 27837 tablet_bootstrap.cc:492] T 379dbe6222c0424fba410e06046e625d P 014a92bf31d243febf2877d5869b6138: No bootstrap required, opened a new log
I20260810 23:26:11.941183 27837 ts_tablet_manager.cc:1397] T 379dbe6222c0424fba410e06046e625d P 014a92bf31d243febf2877d5869b6138: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.006s
I20260810 23:26:11.941815 27836 tablet_bootstrap.cc:492] T 379dbe6222c0424fba410e06046e625d P f46de26a8c5f4921a158eca10ad8970e: No bootstrap required, opened a new log
I20260810 23:26:11.941982 27836 ts_tablet_manager.cc:1397] T 379dbe6222c0424fba410e06046e625d P f46de26a8c5f4921a158eca10ad8970e: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.004s
I20260810 23:26:11.942353 27837 raft_consensus.cc:348] T 379dbe6222c0424fba410e06046e625d P 014a92bf31d243febf2877d5869b6138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:11.942688 27837 raft_consensus.cc:374] T 379dbe6222c0424fba410e06046e625d P 014a92bf31d243febf2877d5869b6138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:26:11.942842 27837 raft_consensus.cc:729] T 379dbe6222c0424fba410e06046e625d P 014a92bf31d243febf2877d5869b6138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014a92bf31d243febf2877d5869b6138, State: Initialized, Role: FOLLOWER
I20260810 23:26:11.942812 27836 raft_consensus.cc:348] T 379dbe6222c0424fba410e06046e625d P f46de26a8c5f4921a158eca10ad8970e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:11.943041 27836 raft_consensus.cc:374] T 379dbe6222c0424fba410e06046e625d P f46de26a8c5f4921a158eca10ad8970e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:26:11.943132 27836 raft_consensus.cc:729] T 379dbe6222c0424fba410e06046e625d P f46de26a8c5f4921a158eca10ad8970e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46de26a8c5f4921a158eca10ad8970e, State: Initialized, Role: FOLLOWER
I20260810 23:26:11.943377 27837 consensus_queue.cc:260] T 379dbe6222c0424fba410e06046e625d P 014a92bf31d243febf2877d5869b6138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:11.944388 27831 tablet_bootstrap.cc:492] T 379dbe6222c0424fba410e06046e625d P d45feef75e5b43d0855345b5de047874: No bootstrap required, opened a new log
I20260810 23:26:11.944562 27831 ts_tablet_manager.cc:1397] T 379dbe6222c0424fba410e06046e625d P d45feef75e5b43d0855345b5de047874: Time spent bootstrapping tablet: real 0.038s user 0.008s sys 0.004s
I20260810 23:26:11.945366 27831 raft_consensus.cc:348] T 379dbe6222c0424fba410e06046e625d P d45feef75e5b43d0855345b5de047874 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:11.945621 27831 raft_consensus.cc:374] T 379dbe6222c0424fba410e06046e625d P d45feef75e5b43d0855345b5de047874 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:26:11.945711 27831 raft_consensus.cc:729] T 379dbe6222c0424fba410e06046e625d P d45feef75e5b43d0855345b5de047874 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d45feef75e5b43d0855345b5de047874, State: Initialized, Role: FOLLOWER
I20260810 23:26:11.946002 27831 consensus_queue.cc:260] T 379dbe6222c0424fba410e06046e625d P d45feef75e5b43d0855345b5de047874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:11.947072 27831 ts_tablet_manager.cc:1428] T 379dbe6222c0424fba410e06046e625d P d45feef75e5b43d0855345b5de047874: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:26:11.950680 27875 ts_tablet_manager.cc:1918] T d16193a9ece14808887bb350c2840a18 P d45feef75e5b43d0855345b5de047874: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:26:11.950861 27875 log.cc:1196] T d16193a9ece14808887bb350c2840a18 P d45feef75e5b43d0855345b5de047874: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-0/wal/wals/d16193a9ece14808887bb350c2840a18
I20260810 23:26:11.951438 27875 ts_tablet_manager.cc:1939] T d16193a9ece14808887bb350c2840a18 P d45feef75e5b43d0855345b5de047874: Deleting consensus metadata
I20260810 23:26:11.952221 27837 ts_tablet_manager.cc:1428] T 379dbe6222c0424fba410e06046e625d P 014a92bf31d243febf2877d5869b6138: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20260810 23:26:11.953390 26953 catalog_manager.cc:4841] TS d45feef75e5b43d0855345b5de047874 (127.14.175.65:46357): tablet d16193a9ece14808887bb350c2840a18 (table test-table [id=4a1330d780cb417b8d141f5c56a7e49c]) successfully deleted
I20260810 23:26:11.953353 27836 consensus_queue.cc:260] T 379dbe6222c0424fba410e06046e625d P f46de26a8c5f4921a158eca10ad8970e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:11.954618 27836 ts_tablet_manager.cc:1428] T 379dbe6222c0424fba410e06046e625d P f46de26a8c5f4921a158eca10ad8970e: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20260810 23:26:11.983042 27847 raft_consensus.cc:482] T 379dbe6222c0424fba410e06046e625d P 014a92bf31d243febf2877d5869b6138 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:26:11.983364 27847 raft_consensus.cc:504] T 379dbe6222c0424fba410e06046e625d P 014a92bf31d243febf2877d5869b6138 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:11.984210 27847 leader_election.cc:288] T 379dbe6222c0424fba410e06046e625d P 014a92bf31d243febf2877d5869b6138 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d45feef75e5b43d0855345b5de047874 (127.14.175.65:46357), f46de26a8c5f4921a158eca10ad8970e (127.14.175.66:46177)
I20260810 23:26:11.984795 27227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "379dbe6222c0424fba410e06046e625d" candidate_uuid: "014a92bf31d243febf2877d5869b6138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d45feef75e5b43d0855345b5de047874" is_pre_election: true
I20260810 23:26:11.985051 27227 raft_consensus.cc:2454] T 379dbe6222c0424fba410e06046e625d P d45feef75e5b43d0855345b5de047874 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 014a92bf31d243febf2877d5869b6138 in term 0.
I20260810 23:26:11.984958 27475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "379dbe6222c0424fba410e06046e625d" candidate_uuid: "014a92bf31d243febf2877d5869b6138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46de26a8c5f4921a158eca10ad8970e" is_pre_election: true
I20260810 23:26:11.985282 27475 raft_consensus.cc:2454] T 379dbe6222c0424fba410e06046e625d P f46de26a8c5f4921a158eca10ad8970e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 014a92bf31d243febf2877d5869b6138 in term 0.
I20260810 23:26:11.985561 27591 leader_election.cc:302] T 379dbe6222c0424fba410e06046e625d P 014a92bf31d243febf2877d5869b6138 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 014a92bf31d243febf2877d5869b6138, d45feef75e5b43d0855345b5de047874; no voters:
I20260810 23:26:11.985918 27847 raft_consensus.cc:2781] T 379dbe6222c0424fba410e06046e625d P 014a92bf31d243febf2877d5869b6138 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:26:11.986057 27847 raft_consensus.cc:482] T 379dbe6222c0424fba410e06046e625d P 014a92bf31d243febf2877d5869b6138 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:26:11.986177 27847 raft_consensus.cc:3037] T 379dbe6222c0424fba410e06046e625d P 014a92bf31d243febf2877d5869b6138 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:26:11.988533 27847 raft_consensus.cc:504] T 379dbe6222c0424fba410e06046e625d P 014a92bf31d243febf2877d5869b6138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:11.989225 27847 leader_election.cc:288] T 379dbe6222c0424fba410e06046e625d P 014a92bf31d243febf2877d5869b6138 [CANDIDATE]: Term 1 election: Requested vote from peers d45feef75e5b43d0855345b5de047874 (127.14.175.65:46357), f46de26a8c5f4921a158eca10ad8970e (127.14.175.66:46177)
I20260810 23:26:11.989871 27227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "379dbe6222c0424fba410e06046e625d" candidate_uuid: "014a92bf31d243febf2877d5869b6138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d45feef75e5b43d0855345b5de047874"
I20260810 23:26:11.989895 27475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "379dbe6222c0424fba410e06046e625d" candidate_uuid: "014a92bf31d243febf2877d5869b6138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46de26a8c5f4921a158eca10ad8970e"
I20260810 23:26:11.990087 27227 raft_consensus.cc:3037] T 379dbe6222c0424fba410e06046e625d P d45feef75e5b43d0855345b5de047874 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:26:11.990125 27475 raft_consensus.cc:3037] T 379dbe6222c0424fba410e06046e625d P f46de26a8c5f4921a158eca10ad8970e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:26:11.992686 27227 raft_consensus.cc:2454] T 379dbe6222c0424fba410e06046e625d P d45feef75e5b43d0855345b5de047874 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 014a92bf31d243febf2877d5869b6138 in term 1.
I20260810 23:26:11.993206 27591 leader_election.cc:302] T 379dbe6222c0424fba410e06046e625d P 014a92bf31d243febf2877d5869b6138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 014a92bf31d243febf2877d5869b6138, d45feef75e5b43d0855345b5de047874; no voters:
I20260810 23:26:11.993194 27475 raft_consensus.cc:2454] T 379dbe6222c0424fba410e06046e625d P f46de26a8c5f4921a158eca10ad8970e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 014a92bf31d243febf2877d5869b6138 in term 1.
I20260810 23:26:11.993853 27847 raft_consensus.cc:2781] T 379dbe6222c0424fba410e06046e625d P 014a92bf31d243febf2877d5869b6138 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:26:11.994035 27847 raft_consensus.cc:686] T 379dbe6222c0424fba410e06046e625d P 014a92bf31d243febf2877d5869b6138 [term 1 LEADER]: Becoming Leader. State: Replica: 014a92bf31d243febf2877d5869b6138, State: Running, Role: LEADER
I20260810 23:26:11.994311 27847 consensus_queue.cc:237] T 379dbe6222c0424fba410e06046e625d P 014a92bf31d243febf2877d5869b6138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:11.997328 26978 catalog_manager.cc:5495] T 379dbe6222c0424fba410e06046e625d P 014a92bf31d243febf2877d5869b6138 reported cstate change: term changed from 0 to 1, leader changed from <none> to 014a92bf31d243febf2877d5869b6138 (127.14.175.67). New cstate: current_term: 1 leader_uuid: "014a92bf31d243febf2877d5869b6138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } health_report { overall_health: HEALTHY } } }
Aug 10 23:26:12 dist-test-slave-rmhg krb5kdc[26900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404369, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.126@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM
I20260810 23:26:12.071723 27604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:12.056053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40250 (local address 127.14.175.126:41621)
0810 23:26:12.056326 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:12.056334 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:12.056882 (+ 548us) server_negotiation.cc:407] Connection header received
0810 23:26:12.057062 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:12.057066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:12.057145 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:12.057264 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:12.058880 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.059636 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.061599 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.062037 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.062171 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:12.065823 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:12.065828 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:12.065831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:12.065862 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:12.068624 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:12.069529 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:12.069536 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:12.069540 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:12.069704 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:12.070085 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:12.070091 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:12.070095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:12.070443 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:12.070602 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:12.071048 (+ 446us) server_negotiation.cc:299] Negotiation successful
0810 23:26:12.071379 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":112,"server-negotiator.queue_time_us":121}
I20260810 23:26:12.071744 27902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:12.056020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:41621 (local address 127.0.0.1:40250)
0810 23:26:12.056749 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:12.056773 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:12.056947 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:12.057320 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:12.057325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:12.058346 (+ 1021us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:12.058706 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:12.058720 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.059778 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.059785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:12.061452 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:12.061469 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.062229 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.062239 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:12.062379 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:12.063058 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:12.063077 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:12.065638 (+ 2561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:12.068761 (+ 3123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:12.068777 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:12.068786 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:12.069416 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:12.069815 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:12.069820 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:12.069824 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:12.069977 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:12.070594 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:12.070610 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:12.070790 (+ 180us) client_negotiation.cc:769] Sending connection context
0810 23:26:12.070931 (+ 141us) client_negotiation.cc:240] Negotiation successful
0810 23:26:12.071234 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":372,"threads_started":1}
I20260810 23:26:12.077023 26978 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:40250:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 23:26:12.078177 26978 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:26:12.085881 27201 tablet_service.cc:1467] Processing CreateTablet for tablet 12ec31a54d0948d99b00b18d73b589f8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9917c68ed5b440abac13d02bc000999f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:26:12.086447 27201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12ec31a54d0948d99b00b18d73b589f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:26:12.089795 27663 tablet_service.cc:1467] Processing CreateTablet for tablet 12ec31a54d0948d99b00b18d73b589f8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9917c68ed5b440abac13d02bc000999f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:26:12.090488 27663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12ec31a54d0948d99b00b18d73b589f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:26:12.090801 27436 tablet_service.cc:1467] Processing CreateTablet for tablet 12ec31a54d0948d99b00b18d73b589f8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9917c68ed5b440abac13d02bc000999f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:26:12.091382 27436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12ec31a54d0948d99b00b18d73b589f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:26:12.092366 27831 tablet_bootstrap.cc:492] T 12ec31a54d0948d99b00b18d73b589f8 P d45feef75e5b43d0855345b5de047874: Bootstrap starting.
I20260810 23:26:12.094169 27831 tablet_bootstrap.cc:654] T 12ec31a54d0948d99b00b18d73b589f8 P d45feef75e5b43d0855345b5de047874: Neither blocks nor log segments found. Creating new log.
I20260810 23:26:12.098367 27837 tablet_bootstrap.cc:492] T 12ec31a54d0948d99b00b18d73b589f8 P 014a92bf31d243febf2877d5869b6138: Bootstrap starting.
I20260810 23:26:12.100311 27837 tablet_bootstrap.cc:654] T 12ec31a54d0948d99b00b18d73b589f8 P 014a92bf31d243febf2877d5869b6138: Neither blocks nor log segments found. Creating new log.
I20260810 23:26:12.101398 27836 tablet_bootstrap.cc:492] T 12ec31a54d0948d99b00b18d73b589f8 P f46de26a8c5f4921a158eca10ad8970e: Bootstrap starting.
I20260810 23:26:12.102790 27831 tablet_bootstrap.cc:492] T 12ec31a54d0948d99b00b18d73b589f8 P d45feef75e5b43d0855345b5de047874: No bootstrap required, opened a new log
I20260810 23:26:12.102968 27831 ts_tablet_manager.cc:1397] T 12ec31a54d0948d99b00b18d73b589f8 P d45feef75e5b43d0855345b5de047874: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260810 23:26:12.103165 27836 tablet_bootstrap.cc:654] T 12ec31a54d0948d99b00b18d73b589f8 P f46de26a8c5f4921a158eca10ad8970e: Neither blocks nor log segments found. Creating new log.
I20260810 23:26:12.103773 27831 raft_consensus.cc:348] T 12ec31a54d0948d99b00b18d73b589f8 P d45feef75e5b43d0855345b5de047874 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:12.104019 27831 raft_consensus.cc:374] T 12ec31a54d0948d99b00b18d73b589f8 P d45feef75e5b43d0855345b5de047874 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:26:12.104111 27831 raft_consensus.cc:729] T 12ec31a54d0948d99b00b18d73b589f8 P d45feef75e5b43d0855345b5de047874 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d45feef75e5b43d0855345b5de047874, State: Initialized, Role: FOLLOWER
I20260810 23:26:12.104418 27831 consensus_queue.cc:260] T 12ec31a54d0948d99b00b18d73b589f8 P d45feef75e5b43d0855345b5de047874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:12.104909 27843 tablet_replica.cc:439] T 12ec31a54d0948d99b00b18d73b589f8 P d45feef75e5b43d0855345b5de047874: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } } }
I20260810 23:26:12.105325 27843 tablet_replica.cc:442] T 12ec31a54d0948d99b00b18d73b589f8 P d45feef75e5b43d0855345b5de047874: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:26:12.106658 27831 ts_tablet_manager.cc:1428] T 12ec31a54d0948d99b00b18d73b589f8 P d45feef75e5b43d0855345b5de047874: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260810 23:26:12.108361 27836 tablet_bootstrap.cc:492] T 12ec31a54d0948d99b00b18d73b589f8 P f46de26a8c5f4921a158eca10ad8970e: No bootstrap required, opened a new log
I20260810 23:26:12.108629 27836 ts_tablet_manager.cc:1397] T 12ec31a54d0948d99b00b18d73b589f8 P f46de26a8c5f4921a158eca10ad8970e: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260810 23:26:12.109525 27836 raft_consensus.cc:348] T 12ec31a54d0948d99b00b18d73b589f8 P f46de26a8c5f4921a158eca10ad8970e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:12.109917 27836 raft_consensus.cc:374] T 12ec31a54d0948d99b00b18d73b589f8 P f46de26a8c5f4921a158eca10ad8970e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:26:12.110067 27836 raft_consensus.cc:729] T 12ec31a54d0948d99b00b18d73b589f8 P f46de26a8c5f4921a158eca10ad8970e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46de26a8c5f4921a158eca10ad8970e, State: Initialized, Role: FOLLOWER
I20260810 23:26:12.110442 27836 consensus_queue.cc:260] T 12ec31a54d0948d99b00b18d73b589f8 P f46de26a8c5f4921a158eca10ad8970e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:12.110877 27842 tablet_replica.cc:439] T 12ec31a54d0948d99b00b18d73b589f8 P f46de26a8c5f4921a158eca10ad8970e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } } }
I20260810 23:26:12.111277 27842 tablet_replica.cc:442] T 12ec31a54d0948d99b00b18d73b589f8 P f46de26a8c5f4921a158eca10ad8970e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:26:12.112311 27836 ts_tablet_manager.cc:1428] T 12ec31a54d0948d99b00b18d73b589f8 P f46de26a8c5f4921a158eca10ad8970e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260810 23:26:12.113865 27837 tablet_bootstrap.cc:492] T 12ec31a54d0948d99b00b18d73b589f8 P 014a92bf31d243febf2877d5869b6138: No bootstrap required, opened a new log
I20260810 23:26:12.114048 27837 ts_tablet_manager.cc:1397] T 12ec31a54d0948d99b00b18d73b589f8 P 014a92bf31d243febf2877d5869b6138: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.008s
I20260810 23:26:12.114931 27837 raft_consensus.cc:348] T 12ec31a54d0948d99b00b18d73b589f8 P 014a92bf31d243febf2877d5869b6138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:12.115206 27837 raft_consensus.cc:374] T 12ec31a54d0948d99b00b18d73b589f8 P 014a92bf31d243febf2877d5869b6138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:26:12.115303 27837 raft_consensus.cc:729] T 12ec31a54d0948d99b00b18d73b589f8 P 014a92bf31d243febf2877d5869b6138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014a92bf31d243febf2877d5869b6138, State: Initialized, Role: FOLLOWER
I20260810 23:26:12.115582 27837 consensus_queue.cc:260] T 12ec31a54d0948d99b00b18d73b589f8 P 014a92bf31d243febf2877d5869b6138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:12.116285 27837 ts_tablet_manager.cc:1428] T 12ec31a54d0948d99b00b18d73b589f8 P 014a92bf31d243febf2877d5869b6138: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260810 23:26:12.116083 27847 tablet_replica.cc:439] T 12ec31a54d0948d99b00b18d73b589f8 P 014a92bf31d243febf2877d5869b6138: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } } }
I20260810 23:26:12.116468 27847 tablet_replica.cc:442] T 12ec31a54d0948d99b00b18d73b589f8 P 014a92bf31d243febf2877d5869b6138: This TxnStatusTablet replica's current role is: FOLLOWER
W20260810 23:26:12.139740 27324 tablet.cc:2369] T 12ec31a54d0948d99b00b18d73b589f8 P d45feef75e5b43d0855345b5de047874: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:26:12.180125 27842 raft_consensus.cc:482] T 12ec31a54d0948d99b00b18d73b589f8 P f46de26a8c5f4921a158eca10ad8970e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:26:12.180325 27842 raft_consensus.cc:504] T 12ec31a54d0948d99b00b18d73b589f8 P f46de26a8c5f4921a158eca10ad8970e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:12.181950 27842 leader_election.cc:288] T 12ec31a54d0948d99b00b18d73b589f8 P f46de26a8c5f4921a158eca10ad8970e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d45feef75e5b43d0855345b5de047874 (127.14.175.65:46357), 014a92bf31d243febf2877d5869b6138 (127.14.175.67:40495)
I20260810 23:26:12.190413 27820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:12.182766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:56635 (local address 127.14.175.65:46357)
0810 23:26:12.182942 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:12.182948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:12.183778 (+ 830us) server_negotiation.cc:407] Connection header received
0810 23:26:12.183954 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:12.183960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:12.184039 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:12.184136 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:26:12.184925 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.186037 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.188071 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.189210 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.189390 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:12.189471 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:12.190028 (+ 557us) server_negotiation.cc:299] Negotiation successful
0810 23:26:12.190167 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 23:26:12.190825 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:12.182391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:55239 (local address 127.14.175.67:40495)
0810 23:26:12.182589 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:12.182597 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:12.183206 (+ 609us) server_negotiation.cc:407] Connection header received
0810 23:26:12.183534 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:12.183540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:12.183621 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:12.183676 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:26:12.184488 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.185653 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.188467 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.189781 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.189929 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:12.190007 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:12.190399 (+ 392us) server_negotiation.cc:299] Negotiation successful
0810 23:26:12.190546 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 23:26:12.191247 27906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:12.182290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:46357 (local address 127.14.175.66:56635)
0810 23:26:12.183640 (+ 1350us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:12.183660 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:12.183814 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:12.184201 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:12.184206 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:12.184235 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:26:12.184706 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:12.184721 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.186163 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.186168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:12.187902 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:12.187916 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.189649 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.189656 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:12.189826 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:12.189862 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 23:26:12.190637 (+ 775us) client_negotiation.cc:240] Negotiation successful
0810 23:26:12.190781 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1223,"mutex_wait_us":28,"thread_start_us":731,"threads_started":1}
I20260810 23:26:12.191738 27908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:12.182251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:40495 (local address 127.14.175.66:55239)
0810 23:26:12.183066 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:12.183094 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:12.183347 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:12.183882 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:12.183888 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:12.183917 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:26:12.184314 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:12.184325 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.185806 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.185811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:12.188268 (+ 2457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:12.188285 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.190009 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.190017 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:12.190210 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:12.190245 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 23:26:12.190349 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 23:26:12.190825 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"mutex_wait_us":26,"thread_start_us":1120,"threads_started":1}
I20260810 23:26:12.191887 27227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ec31a54d0948d99b00b18d73b589f8" candidate_uuid: "f46de26a8c5f4921a158eca10ad8970e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d45feef75e5b43d0855345b5de047874" is_pre_election: true
I20260810 23:26:12.192205 27227 raft_consensus.cc:2454] T 12ec31a54d0948d99b00b18d73b589f8 P d45feef75e5b43d0855345b5de047874 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46de26a8c5f4921a158eca10ad8970e in term 0.
I20260810 23:26:12.192734 27354 leader_election.cc:302] T 12ec31a54d0948d99b00b18d73b589f8 P f46de26a8c5f4921a158eca10ad8970e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d45feef75e5b43d0855345b5de047874, f46de26a8c5f4921a158eca10ad8970e; no voters:
I20260810 23:26:12.193267 27842 raft_consensus.cc:2781] T 12ec31a54d0948d99b00b18d73b589f8 P f46de26a8c5f4921a158eca10ad8970e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:26:12.193178 27718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ec31a54d0948d99b00b18d73b589f8" candidate_uuid: "f46de26a8c5f4921a158eca10ad8970e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014a92bf31d243febf2877d5869b6138" is_pre_election: true
I20260810 23:26:12.193393 27842 raft_consensus.cc:482] T 12ec31a54d0948d99b00b18d73b589f8 P f46de26a8c5f4921a158eca10ad8970e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:26:12.193517 27842 raft_consensus.cc:3037] T 12ec31a54d0948d99b00b18d73b589f8 P f46de26a8c5f4921a158eca10ad8970e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:26:12.193655 27718 raft_consensus.cc:2454] T 12ec31a54d0948d99b00b18d73b589f8 P 014a92bf31d243febf2877d5869b6138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46de26a8c5f4921a158eca10ad8970e in term 0.
I20260810 23:26:12.196803 27842 raft_consensus.cc:504] T 12ec31a54d0948d99b00b18d73b589f8 P f46de26a8c5f4921a158eca10ad8970e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:12.197613 27842 leader_election.cc:288] T 12ec31a54d0948d99b00b18d73b589f8 P f46de26a8c5f4921a158eca10ad8970e [CANDIDATE]: Term 1 election: Requested vote from peers d45feef75e5b43d0855345b5de047874 (127.14.175.65:46357), 014a92bf31d243febf2877d5869b6138 (127.14.175.67:40495)
I20260810 23:26:12.198163 27227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ec31a54d0948d99b00b18d73b589f8" candidate_uuid: "f46de26a8c5f4921a158eca10ad8970e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d45feef75e5b43d0855345b5de047874"
I20260810 23:26:12.198429 27227 raft_consensus.cc:3037] T 12ec31a54d0948d99b00b18d73b589f8 P d45feef75e5b43d0855345b5de047874 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:26:12.201462 27718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ec31a54d0948d99b00b18d73b589f8" candidate_uuid: "f46de26a8c5f4921a158eca10ad8970e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014a92bf31d243febf2877d5869b6138"
I20260810 23:26:12.201620 27227 raft_consensus.cc:2454] T 12ec31a54d0948d99b00b18d73b589f8 P d45feef75e5b43d0855345b5de047874 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46de26a8c5f4921a158eca10ad8970e in term 1.
I20260810 23:26:12.201709 27718 raft_consensus.cc:3037] T 12ec31a54d0948d99b00b18d73b589f8 P 014a92bf31d243febf2877d5869b6138 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:26:12.202237 27354 leader_election.cc:302] T 12ec31a54d0948d99b00b18d73b589f8 P f46de26a8c5f4921a158eca10ad8970e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d45feef75e5b43d0855345b5de047874, f46de26a8c5f4921a158eca10ad8970e; no voters:
I20260810 23:26:12.202621 27842 raft_consensus.cc:2781] T 12ec31a54d0948d99b00b18d73b589f8 P f46de26a8c5f4921a158eca10ad8970e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:26:12.203159 27842 raft_consensus.cc:686] T 12ec31a54d0948d99b00b18d73b589f8 P f46de26a8c5f4921a158eca10ad8970e [term 1 LEADER]: Becoming Leader. State: Replica: f46de26a8c5f4921a158eca10ad8970e, State: Running, Role: LEADER
I20260810 23:26:12.203915 27842 consensus_queue.cc:237] T 12ec31a54d0948d99b00b18d73b589f8 P f46de26a8c5f4921a158eca10ad8970e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:12.204417 27718 raft_consensus.cc:2454] T 12ec31a54d0948d99b00b18d73b589f8 P 014a92bf31d243febf2877d5869b6138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46de26a8c5f4921a158eca10ad8970e in term 1.
I20260810 23:26:12.206373 27911 tablet_replica.cc:439] T 12ec31a54d0948d99b00b18d73b589f8 P f46de26a8c5f4921a158eca10ad8970e: TxnStatusTablet state changed. Reason: New leader f46de26a8c5f4921a158eca10ad8970e. Latest consensus state: current_term: 1 leader_uuid: "f46de26a8c5f4921a158eca10ad8970e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } } }
I20260810 23:26:12.206889 27911 tablet_replica.cc:442] T 12ec31a54d0948d99b00b18d73b589f8 P f46de26a8c5f4921a158eca10ad8970e: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:26:12.207682 27913 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 23:26:12.207914 27913 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 23:26:12.208496 26978 catalog_manager.cc:5495] T 12ec31a54d0948d99b00b18d73b589f8 P f46de26a8c5f4921a158eca10ad8970e reported cstate change: term changed from 0 to 1, leader changed from <none> to f46de26a8c5f4921a158eca10ad8970e (127.14.175.66). New cstate: current_term: 1 leader_uuid: "f46de26a8c5f4921a158eca10ad8970e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } health_report { overall_health: UNKNOWN } } }
I20260810 23:26:12.235827 27826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:12.227959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41184 (local address 127.14.175.66:46177)
0810 23:26:12.228173 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:12.228179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:12.228276 (+ 97us) server_negotiation.cc:407] Connection header received
0810 23:26:12.228600 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:12.228605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:12.228736 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:12.228919 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:26:12.229465 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.230724 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.232665 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.233409 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.233585 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:12.234228 (+ 643us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:26:12.235044 (+ 816us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:26:12.235222 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:12.235374 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 23:26:12.235481 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 23:26:12.235857 27902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:12.227765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:46177 (local address 127.0.0.1:41184)
0810 23:26:12.228169 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:12.228191 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:12.228362 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:12.228948 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:12.228953 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:12.228966 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:26:12.229237 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:12.229252 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.230896 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.230902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:12.232478 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:12.232497 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.233638 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.233644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:12.233882 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:12.233989 (+ 107us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:26:12.235194 (+ 1205us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:26:12.235229 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 23:26:12.235336 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 23:26:12.235524 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205}
I20260810 23:26:12.241475 27227 raft_consensus.cc:1261] T 12ec31a54d0948d99b00b18d73b589f8 P d45feef75e5b43d0855345b5de047874 [term 1 FOLLOWER]: Refusing update from remote peer f46de26a8c5f4921a158eca10ad8970e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:26:12.241720 27718 raft_consensus.cc:1261] T 12ec31a54d0948d99b00b18d73b589f8 P 014a92bf31d243febf2877d5869b6138 [term 1 FOLLOWER]: Refusing update from remote peer f46de26a8c5f4921a158eca10ad8970e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:26:12.242524 27842 consensus_queue.cc:1035] T 12ec31a54d0948d99b00b18d73b589f8 P f46de26a8c5f4921a158eca10ad8970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:26:12.243069 27911 consensus_queue.cc:1035] T 12ec31a54d0948d99b00b18d73b589f8 P f46de26a8c5f4921a158eca10ad8970e [LEADER]: Connected to new peer: Peer: permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:26:12.257736 27843 tablet_replica.cc:439] T 12ec31a54d0948d99b00b18d73b589f8 P d45feef75e5b43d0855345b5de047874: TxnStatusTablet state changed. Reason: New leader f46de26a8c5f4921a158eca10ad8970e. Latest consensus state: current_term: 1 leader_uuid: "f46de26a8c5f4921a158eca10ad8970e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } } }
I20260810 23:26:12.258065 27843 tablet_replica.cc:442] T 12ec31a54d0948d99b00b18d73b589f8 P d45feef75e5b43d0855345b5de047874: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:26:12.258699 27847 tablet_replica.cc:439] T 12ec31a54d0948d99b00b18d73b589f8 P 014a92bf31d243febf2877d5869b6138: TxnStatusTablet state changed. Reason: New leader f46de26a8c5f4921a158eca10ad8970e. Latest consensus state: current_term: 1 leader_uuid: "f46de26a8c5f4921a158eca10ad8970e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } } }
I20260810 23:26:12.258992 27847 tablet_replica.cc:442] T 12ec31a54d0948d99b00b18d73b589f8 P 014a92bf31d243febf2877d5869b6138: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:26:12.264595 27842 tablet_replica.cc:439] T 12ec31a54d0948d99b00b18d73b589f8 P f46de26a8c5f4921a158eca10ad8970e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f46de26a8c5f4921a158eca10ad8970e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } } }
I20260810 23:26:12.264940 27842 tablet_replica.cc:442] T 12ec31a54d0948d99b00b18d73b589f8 P f46de26a8c5f4921a158eca10ad8970e: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:26:12.277945 27911 tablet_replica.cc:439] T 12ec31a54d0948d99b00b18d73b589f8 P f46de26a8c5f4921a158eca10ad8970e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f46de26a8c5f4921a158eca10ad8970e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } } }
I20260810 23:26:12.278303 27911 tablet_replica.cc:442] T 12ec31a54d0948d99b00b18d73b589f8 P f46de26a8c5f4921a158eca10ad8970e: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:26:12.280512 27847 tablet_replica.cc:439] T 12ec31a54d0948d99b00b18d73b589f8 P 014a92bf31d243febf2877d5869b6138: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f46de26a8c5f4921a158eca10ad8970e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } } }
I20260810 23:26:12.280810 27847 tablet_replica.cc:442] T 12ec31a54d0948d99b00b18d73b589f8 P 014a92bf31d243febf2877d5869b6138: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:26:12.287024 27843 tablet_replica.cc:439] T 12ec31a54d0948d99b00b18d73b589f8 P d45feef75e5b43d0855345b5de047874: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f46de26a8c5f4921a158eca10ad8970e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } } }
I20260810 23:26:12.287333 27843 tablet_replica.cc:442] T 12ec31a54d0948d99b00b18d73b589f8 P d45feef75e5b43d0855345b5de047874: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:26:12.303906 27826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:12.295807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:40439 (local address 127.14.175.66:46177)
0810 23:26:12.296011 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:12.296019 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:12.296693 (+ 674us) server_negotiation.cc:407] Connection header received
0810 23:26:12.296909 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:12.296916 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:12.297028 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:12.297183 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:26:12.297827 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.299030 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.301462 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.302195 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.302350 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:12.303000 (+ 650us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:26:12.303182 (+ 182us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:26:12.303288 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:12.303477 (+ 189us) server_negotiation.cc:299] Negotiation successful
0810 23:26:12.303580 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 23:26:12.304039 27931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:12.295829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:46177 (local address 127.14.175.67:40439)
0810 23:26:12.296556 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:12.296576 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:12.296747 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:12.297221 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:12.297227 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:12.297241 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:26:12.297626 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:12.297641 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.299215 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.299222 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:12.301247 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:12.301268 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.302431 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.302439 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:12.302653 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:12.302759 (+ 106us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:26:12.303305 (+ 546us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:26:12.303318 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 23:26:12.303410 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 23:26:12.303583 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":349,"threads_started":1}
I20260810 23:26:12.317632 27227 raft_consensus.cc:1261] T 379dbe6222c0424fba410e06046e625d P d45feef75e5b43d0855345b5de047874 [term 1 FOLLOWER]: Refusing update from remote peer 014a92bf31d243febf2877d5869b6138: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:26:12.317679 27475 raft_consensus.cc:1261] T 379dbe6222c0424fba410e06046e625d P f46de26a8c5f4921a158eca10ad8970e [term 1 FOLLOWER]: Refusing update from remote peer 014a92bf31d243febf2877d5869b6138: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:26:12.318388 27844 consensus_queue.cc:1035] T 379dbe6222c0424fba410e06046e625d P 014a92bf31d243febf2877d5869b6138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:26:12.318917 27844 consensus_queue.cc:1035] T 379dbe6222c0424fba410e06046e625d P 014a92bf31d243febf2877d5869b6138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:26:12.334020 27853 mvcc.cc:205] Tried to move back new op lower bound from 7317112309005172736 to 7317112307691479040. Current Snapshot: MvccSnapshot[applied={T|T < 7317112309005172736}]
I20260810 23:26:12.402704 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:12.371784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:60461 (local address 127.14.175.67:40495)
0810 23:26:12.371998 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:12.372005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:12.390088 (+ 18083us) server_negotiation.cc:407] Connection header received
0810 23:26:12.390175 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:12.390181 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:12.390295 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:12.390389 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:26:12.391163 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.392281 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.394601 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.395321 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.395464 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:12.401872 (+ 6408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:26:12.402024 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:26:12.402085 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:12.402315 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 23:26:12.402409 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 23:26:12.403506 27942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:12.371437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:40495 (local address 127.14.175.66:60461)
0810 23:26:12.389737 (+ 18300us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:12.389762 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:12.389924 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:12.390576 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:12.390581 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:12.390596 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:26:12.390955 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:12.390968 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.392464 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.392470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:12.394440 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:12.394458 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.401329 (+ 6871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.401336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:12.401550 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:12.401634 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:26:12.402179 (+ 545us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:26:12.402189 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 23:26:12.402907 (+ 718us) client_negotiation.cc:240] Negotiation successful
0810 23:26:12.403073 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18009,"thread_start_us":18569,"threads_started":1}
I20260810 23:26:12.446890 26979 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:26:12.447172 26979 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:26:12.450387 26979 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b274abd910bd4f9d88d4d17b7bf70fdc: Sending DeleteTablet for 3 replicas of tablet 379dbe6222c0424fba410e06046e625d
I20260810 23:26:12.451263 27201 tablet_service.cc:1514] Processing DeleteTablet for tablet 379dbe6222c0424fba410e06046e625d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:26:12 UTC) from {username='oryx', principal='oryx/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:58188
I20260810 23:26:12.451478 27663 tablet_service.cc:1514] Processing DeleteTablet for tablet 379dbe6222c0424fba410e06046e625d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:26:12 UTC) from {username='oryx', principal='oryx/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:54782
I20260810 23:26:12.451560 27875 tablet_replica.cc:330] T 379dbe6222c0424fba410e06046e625d P d45feef75e5b43d0855345b5de047874: stopping tablet replica
I20260810 23:26:12.451781 27875 raft_consensus.cc:2229] T 379dbe6222c0424fba410e06046e625d P d45feef75e5b43d0855345b5de047874 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:26:12.451987 27875 raft_consensus.cc:2258] T 379dbe6222c0424fba410e06046e625d P d45feef75e5b43d0855345b5de047874 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:26:12.453347 27875 ts_tablet_manager.cc:1905] T 379dbe6222c0424fba410e06046e625d P d45feef75e5b43d0855345b5de047874: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:26:12.457417 27436 tablet_service.cc:1514] Processing DeleteTablet for tablet 379dbe6222c0424fba410e06046e625d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:26:12 UTC) from {username='oryx', principal='oryx/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:41168
I20260810 23:26:12.460683 27875 ts_tablet_manager.cc:1918] T 379dbe6222c0424fba410e06046e625d P d45feef75e5b43d0855345b5de047874: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:26:12.460903 27875 log.cc:1196] T 379dbe6222c0424fba410e06046e625d P d45feef75e5b43d0855345b5de047874: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-0/wal/wals/379dbe6222c0424fba410e06046e625d
I20260810 23:26:12.461407 27875 ts_tablet_manager.cc:1939] T 379dbe6222c0424fba410e06046e625d P d45feef75e5b43d0855345b5de047874: Deleting consensus metadata
I20260810 23:26:12.462760 26953 catalog_manager.cc:4841] TS d45feef75e5b43d0855345b5de047874 (127.14.175.65:46357): tablet 379dbe6222c0424fba410e06046e625d (table test-table [id=6f6c84df7d6d4f49b79c5cc822386596]) successfully deleted
I20260810 23:26:12.463609 27947 tablet_replica.cc:330] T 379dbe6222c0424fba410e06046e625d P f46de26a8c5f4921a158eca10ad8970e: stopping tablet replica
I20260810 23:26:12.463871 27947 raft_consensus.cc:2229] T 379dbe6222c0424fba410e06046e625d P f46de26a8c5f4921a158eca10ad8970e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:26:12.464176 27947 raft_consensus.cc:2258] T 379dbe6222c0424fba410e06046e625d P f46de26a8c5f4921a158eca10ad8970e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:26:12.465296 27947 ts_tablet_manager.cc:1905] T 379dbe6222c0424fba410e06046e625d P f46de26a8c5f4921a158eca10ad8970e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:26:12.471233 27945 tablet_replica.cc:330] T 379dbe6222c0424fba410e06046e625d P 014a92bf31d243febf2877d5869b6138: stopping tablet replica
I20260810 23:26:12.471541 27945 raft_consensus.cc:2229] T 379dbe6222c0424fba410e06046e625d P 014a92bf31d243febf2877d5869b6138 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:26:12.472041 27945 raft_consensus.cc:2258] T 379dbe6222c0424fba410e06046e625d P 014a92bf31d243febf2877d5869b6138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:26:12.473110 27947 ts_tablet_manager.cc:1918] T 379dbe6222c0424fba410e06046e625d P f46de26a8c5f4921a158eca10ad8970e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:26:12.473280 27945 ts_tablet_manager.cc:1905] T 379dbe6222c0424fba410e06046e625d P 014a92bf31d243febf2877d5869b6138: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:26:12.473358 27947 log.cc:1196] T 379dbe6222c0424fba410e06046e625d P f46de26a8c5f4921a158eca10ad8970e: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-1/wal/wals/379dbe6222c0424fba410e06046e625d
I20260810 23:26:12.473882 27947 ts_tablet_manager.cc:1939] T 379dbe6222c0424fba410e06046e625d P f46de26a8c5f4921a158eca10ad8970e: Deleting consensus metadata
I20260810 23:26:12.475207 26953 catalog_manager.cc:4841] TS f46de26a8c5f4921a158eca10ad8970e (127.14.175.66:46177): tablet 379dbe6222c0424fba410e06046e625d (table test-table [id=6f6c84df7d6d4f49b79c5cc822386596]) successfully deleted
I20260810 23:26:12.480756 27945 ts_tablet_manager.cc:1918] T 379dbe6222c0424fba410e06046e625d P 014a92bf31d243febf2877d5869b6138: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:26:12.480926 27945 log.cc:1196] T 379dbe6222c0424fba410e06046e625d P 014a92bf31d243febf2877d5869b6138: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-2/wal/wals/379dbe6222c0424fba410e06046e625d
I20260810 23:26:12.481549 27945 ts_tablet_manager.cc:1939] T 379dbe6222c0424fba410e06046e625d P 014a92bf31d243febf2877d5869b6138: Deleting consensus metadata
I20260810 23:26:12.482839 26956 catalog_manager.cc:4841] TS 014a92bf31d243febf2877d5869b6138 (127.14.175.67:40495): tablet 379dbe6222c0424fba410e06046e625d (table test-table [id=6f6c84df7d6d4f49b79c5cc822386596]) successfully deleted
Aug 10 23:26:12 dist-test-slave-rmhg krb5kdc[26900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 23:26:12 dist-test-slave-rmhg krb5kdc[26900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:12 dist-test-slave-rmhg krb5kdc[26900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 23:26:12.537915 27604 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:12.524917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40258 (local address 127.14.175.126:41621)
0810 23:26:12.525144 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:12.525204 (+ 60us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:12.525781 (+ 577us) server_negotiation.cc:407] Connection header received
0810 23:26:12.525963 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:12.525970 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:12.526102 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:12.526206 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:12.528129 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.529393 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.531365 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.532072 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.532224 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:12.537647 (+ 5423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40258: BlockingRecv error: recv got EOF from 127.0.0.1:40258 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 10 23:26:12 dist-test-slave-rmhg krb5kdc[26900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM
I20260810 23:26:12.569689 27604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:12.554176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40272 (local address 127.14.175.126:41621)
0810 23:26:12.554380 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:12.554387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:12.554793 (+ 406us) server_negotiation.cc:407] Connection header received
0810 23:26:12.554974 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:12.554978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:12.555063 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:12.555166 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:12.556527 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.557366 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.558724 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.559162 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.559286 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:12.563827 (+ 4541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:12.563833 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:12.563837 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:12.563882 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:12.566684 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:12.567458 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:12.567463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:12.567466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:12.567635 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:12.568096 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:12.568102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:12.568105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:12.568428 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:12.568721 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:12.569060 (+ 339us) server_negotiation.cc:299] Negotiation successful
0810 23:26:12.569344 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:26:12.573998 26979 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:26:12.574712 26979 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:26:12.582414 27201 tablet_service.cc:1467] Processing CreateTablet for tablet 679ea46a35aa4922bd5b199757fbe6bc (DEFAULT_TABLE table=test-table [id=abd0d89c948f46108bc083c158b411c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:26:12.582775 27663 tablet_service.cc:1467] Processing CreateTablet for tablet 679ea46a35aa4922bd5b199757fbe6bc (DEFAULT_TABLE table=test-table [id=abd0d89c948f46108bc083c158b411c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:26:12.582907 27201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 679ea46a35aa4922bd5b199757fbe6bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:26:12.583231 27663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 679ea46a35aa4922bd5b199757fbe6bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:26:12.587201 27436 tablet_service.cc:1467] Processing CreateTablet for tablet 679ea46a35aa4922bd5b199757fbe6bc (DEFAULT_TABLE table=test-table [id=abd0d89c948f46108bc083c158b411c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:26:12.587709 27436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 679ea46a35aa4922bd5b199757fbe6bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:26:12.589773 27831 tablet_bootstrap.cc:492] T 679ea46a35aa4922bd5b199757fbe6bc P d45feef75e5b43d0855345b5de047874: Bootstrap starting.
I20260810 23:26:12.591233 27837 tablet_bootstrap.cc:492] T 679ea46a35aa4922bd5b199757fbe6bc P 014a92bf31d243febf2877d5869b6138: Bootstrap starting.
I20260810 23:26:12.591635 27831 tablet_bootstrap.cc:654] T 679ea46a35aa4922bd5b199757fbe6bc P d45feef75e5b43d0855345b5de047874: Neither blocks nor log segments found. Creating new log.
I20260810 23:26:12.592862 27837 tablet_bootstrap.cc:654] T 679ea46a35aa4922bd5b199757fbe6bc P 014a92bf31d243febf2877d5869b6138: Neither blocks nor log segments found. Creating new log.
I20260810 23:26:12.595494 27836 tablet_bootstrap.cc:492] T 679ea46a35aa4922bd5b199757fbe6bc P f46de26a8c5f4921a158eca10ad8970e: Bootstrap starting.
I20260810 23:26:12.597308 27836 tablet_bootstrap.cc:654] T 679ea46a35aa4922bd5b199757fbe6bc P f46de26a8c5f4921a158eca10ad8970e: Neither blocks nor log segments found. Creating new log.
I20260810 23:26:12.603416 27837 tablet_bootstrap.cc:492] T 679ea46a35aa4922bd5b199757fbe6bc P 014a92bf31d243febf2877d5869b6138: No bootstrap required, opened a new log
I20260810 23:26:12.603416 27831 tablet_bootstrap.cc:492] T 679ea46a35aa4922bd5b199757fbe6bc P d45feef75e5b43d0855345b5de047874: No bootstrap required, opened a new log
I20260810 23:26:12.603597 27831 ts_tablet_manager.cc:1397] T 679ea46a35aa4922bd5b199757fbe6bc P d45feef75e5b43d0855345b5de047874: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260810 23:26:12.603597 27837 ts_tablet_manager.cc:1397] T 679ea46a35aa4922bd5b199757fbe6bc P 014a92bf31d243febf2877d5869b6138: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260810 23:26:12.604411 27837 raft_consensus.cc:348] T 679ea46a35aa4922bd5b199757fbe6bc P 014a92bf31d243febf2877d5869b6138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:12.604411 27831 raft_consensus.cc:348] T 679ea46a35aa4922bd5b199757fbe6bc P d45feef75e5b43d0855345b5de047874 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:12.604707 27837 raft_consensus.cc:374] T 679ea46a35aa4922bd5b199757fbe6bc P 014a92bf31d243febf2877d5869b6138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:26:12.604707 27831 raft_consensus.cc:374] T 679ea46a35aa4922bd5b199757fbe6bc P d45feef75e5b43d0855345b5de047874 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:26:12.604805 27831 raft_consensus.cc:729] T 679ea46a35aa4922bd5b199757fbe6bc P d45feef75e5b43d0855345b5de047874 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d45feef75e5b43d0855345b5de047874, State: Initialized, Role: FOLLOWER
I20260810 23:26:12.604805 27837 raft_consensus.cc:729] T 679ea46a35aa4922bd5b199757fbe6bc P 014a92bf31d243febf2877d5869b6138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014a92bf31d243febf2877d5869b6138, State: Initialized, Role: FOLLOWER
I20260810 23:26:12.605072 27831 consensus_queue.cc:260] T 679ea46a35aa4922bd5b199757fbe6bc P d45feef75e5b43d0855345b5de047874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:12.605072 27837 consensus_queue.cc:260] T 679ea46a35aa4922bd5b199757fbe6bc P 014a92bf31d243febf2877d5869b6138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:12.605672 27831 ts_tablet_manager.cc:1428] T 679ea46a35aa4922bd5b199757fbe6bc P d45feef75e5b43d0855345b5de047874: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:26:12.606295 27837 ts_tablet_manager.cc:1428] T 679ea46a35aa4922bd5b199757fbe6bc P 014a92bf31d243febf2877d5869b6138: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 23:26:12.608878 27836 tablet_bootstrap.cc:492] T 679ea46a35aa4922bd5b199757fbe6bc P f46de26a8c5f4921a158eca10ad8970e: No bootstrap required, opened a new log
I20260810 23:26:12.609058 27836 ts_tablet_manager.cc:1397] T 679ea46a35aa4922bd5b199757fbe6bc P f46de26a8c5f4921a158eca10ad8970e: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.002s
I20260810 23:26:12.609967 27836 raft_consensus.cc:348] T 679ea46a35aa4922bd5b199757fbe6bc P f46de26a8c5f4921a158eca10ad8970e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:12.610311 27836 raft_consensus.cc:374] T 679ea46a35aa4922bd5b199757fbe6bc P f46de26a8c5f4921a158eca10ad8970e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:26:12.610428 27836 raft_consensus.cc:729] T 679ea46a35aa4922bd5b199757fbe6bc P f46de26a8c5f4921a158eca10ad8970e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46de26a8c5f4921a158eca10ad8970e, State: Initialized, Role: FOLLOWER
I20260810 23:26:12.610697 27836 consensus_queue.cc:260] T 679ea46a35aa4922bd5b199757fbe6bc P f46de26a8c5f4921a158eca10ad8970e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:12.611187 27836 ts_tablet_manager.cc:1428] T 679ea46a35aa4922bd5b199757fbe6bc P f46de26a8c5f4921a158eca10ad8970e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:26:12.727687 27843 raft_consensus.cc:482] T 679ea46a35aa4922bd5b199757fbe6bc P d45feef75e5b43d0855345b5de047874 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:26:12.727932 27843 raft_consensus.cc:504] T 679ea46a35aa4922bd5b199757fbe6bc P d45feef75e5b43d0855345b5de047874 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:12.729444 27843 leader_election.cc:288] T 679ea46a35aa4922bd5b199757fbe6bc P d45feef75e5b43d0855345b5de047874 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46de26a8c5f4921a158eca10ad8970e (127.14.175.66:46177), 014a92bf31d243febf2877d5869b6138 (127.14.175.67:40495)
I20260810 23:26:12.736006 27976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:12.729753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:40495 (local address 127.14.175.65:54013)
0810 23:26:12.730247 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:12.730266 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:12.730396 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:12.730744 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:12.730748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:12.730801 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:26:12.731067 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:12.731080 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.732356 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.732361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:12.733916 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:12.733930 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.735238 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.735244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:12.735416 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:12.735453 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 23:26:12.735577 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 23:26:12.735726 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":253,"threads_started":1}
I20260810 23:26:12.736074 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:12.729912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:54013 (local address 127.14.175.67:40495)
0810 23:26:12.730141 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:12.730146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:12.730371 (+ 225us) server_negotiation.cc:407] Connection header received
0810 23:26:12.730554 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:12.730558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:12.730622 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:12.730700 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:26:12.731256 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.732206 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.734094 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.735079 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.735200 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:12.735255 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:12.735612 (+ 357us) server_negotiation.cc:299] Negotiation successful
0810 23:26:12.735749 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 23:26:12.736610 27977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:12.729753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:46177 (local address 127.14.175.65:57105)
0810 23:26:12.730789 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:12.730804 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:12.730949 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:12.731298 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:12.731308 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:12.731334 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:26:12.731715 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:12.731728 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.732822 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.732826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:12.734399 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:12.734424 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.735759 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.735764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:12.735956 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:12.735989 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 23:26:12.736091 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 23:26:12.736224 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"mutex_wait_us":404,"spinlock_wait_cycles":1280,"thread_start_us":337,"threads_started":1}
I20260810 23:26:12.736676 27718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "679ea46a35aa4922bd5b199757fbe6bc" candidate_uuid: "d45feef75e5b43d0855345b5de047874" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014a92bf31d243febf2877d5869b6138" is_pre_election: true
I20260810 23:26:12.736790 27826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:12.729921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:57105 (local address 127.14.175.66:46177)
0810 23:26:12.730133 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:12.730139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:12.730901 (+ 762us) server_negotiation.cc:407] Connection header received
0810 23:26:12.731081 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:12.731087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:12.731171 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:12.731261 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:26:12.731861 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.732689 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.734572 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.735524 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.735670 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:12.735737 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:12.736401 (+ 664us) server_negotiation.cc:299] Negotiation successful
0810 23:26:12.736546 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 23:26:12.736934 27718 raft_consensus.cc:2454] T 679ea46a35aa4922bd5b199757fbe6bc P 014a92bf31d243febf2877d5869b6138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d45feef75e5b43d0855345b5de047874 in term 0.
I20260810 23:26:12.737380 27475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "679ea46a35aa4922bd5b199757fbe6bc" candidate_uuid: "d45feef75e5b43d0855345b5de047874" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46de26a8c5f4921a158eca10ad8970e" is_pre_election: true
I20260810 23:26:12.737474 27126 leader_election.cc:302] T 679ea46a35aa4922bd5b199757fbe6bc P d45feef75e5b43d0855345b5de047874 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 014a92bf31d243febf2877d5869b6138, d45feef75e5b43d0855345b5de047874; no voters:
I20260810 23:26:12.737653 27475 raft_consensus.cc:2454] T 679ea46a35aa4922bd5b199757fbe6bc P f46de26a8c5f4921a158eca10ad8970e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d45feef75e5b43d0855345b5de047874 in term 0.
I20260810 23:26:12.737859 27843 raft_consensus.cc:2781] T 679ea46a35aa4922bd5b199757fbe6bc P d45feef75e5b43d0855345b5de047874 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:26:12.738005 27843 raft_consensus.cc:482] T 679ea46a35aa4922bd5b199757fbe6bc P d45feef75e5b43d0855345b5de047874 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:26:12.738117 27843 raft_consensus.cc:3037] T 679ea46a35aa4922bd5b199757fbe6bc P d45feef75e5b43d0855345b5de047874 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:26:12.741405 27843 raft_consensus.cc:504] T 679ea46a35aa4922bd5b199757fbe6bc P d45feef75e5b43d0855345b5de047874 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:12.742162 27843 leader_election.cc:288] T 679ea46a35aa4922bd5b199757fbe6bc P d45feef75e5b43d0855345b5de047874 [CANDIDATE]: Term 1 election: Requested vote from peers f46de26a8c5f4921a158eca10ad8970e (127.14.175.66:46177), 014a92bf31d243febf2877d5869b6138 (127.14.175.67:40495)
I20260810 23:26:12.742738 27718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "679ea46a35aa4922bd5b199757fbe6bc" candidate_uuid: "d45feef75e5b43d0855345b5de047874" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014a92bf31d243febf2877d5869b6138"
I20260810 23:26:12.742760 27475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "679ea46a35aa4922bd5b199757fbe6bc" candidate_uuid: "d45feef75e5b43d0855345b5de047874" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46de26a8c5f4921a158eca10ad8970e"
I20260810 23:26:12.742939 27718 raft_consensus.cc:3037] T 679ea46a35aa4922bd5b199757fbe6bc P 014a92bf31d243febf2877d5869b6138 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:26:12.742971 27475 raft_consensus.cc:3037] T 679ea46a35aa4922bd5b199757fbe6bc P f46de26a8c5f4921a158eca10ad8970e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:26:12.745460 27718 raft_consensus.cc:2454] T 679ea46a35aa4922bd5b199757fbe6bc P 014a92bf31d243febf2877d5869b6138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d45feef75e5b43d0855345b5de047874 in term 1.
I20260810 23:26:12.745978 27475 raft_consensus.cc:2454] T 679ea46a35aa4922bd5b199757fbe6bc P f46de26a8c5f4921a158eca10ad8970e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d45feef75e5b43d0855345b5de047874 in term 1.
I20260810 23:26:12.746225 27126 leader_election.cc:302] T 679ea46a35aa4922bd5b199757fbe6bc P d45feef75e5b43d0855345b5de047874 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 014a92bf31d243febf2877d5869b6138, d45feef75e5b43d0855345b5de047874; no voters:
I20260810 23:26:12.746605 27843 raft_consensus.cc:2781] T 679ea46a35aa4922bd5b199757fbe6bc P d45feef75e5b43d0855345b5de047874 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:26:12.747031 27843 raft_consensus.cc:686] T 679ea46a35aa4922bd5b199757fbe6bc P d45feef75e5b43d0855345b5de047874 [term 1 LEADER]: Becoming Leader. State: Replica: d45feef75e5b43d0855345b5de047874, State: Running, Role: LEADER
I20260810 23:26:12.747475 27843 consensus_queue.cc:237] T 679ea46a35aa4922bd5b199757fbe6bc P d45feef75e5b43d0855345b5de047874 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:12.751240 26978 catalog_manager.cc:5495] T 679ea46a35aa4922bd5b199757fbe6bc P d45feef75e5b43d0855345b5de047874 reported cstate change: term changed from 0 to 1, leader changed from <none> to d45feef75e5b43d0855345b5de047874 (127.14.175.65). New cstate: current_term: 1 leader_uuid: "d45feef75e5b43d0855345b5de047874" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } health_report { overall_health: UNKNOWN } } }
I20260810 23:26:12.801015 27983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:12.795204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58194 (local address 127.14.175.65:46357)
0810 23:26:12.795781 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:12.795786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:12.795809 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:26:12.795905 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:12.795910 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:12.795989 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:12.796149 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:26:12.796513 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.797361 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.798711 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.799163 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.799265 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:12.799735 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:26:12.800378 (+ 643us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:26:12.800489 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:12.800579 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 23:26:12.800672 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":313,"threads_started":1}
I20260810 23:26:12.804767 27718 raft_consensus.cc:1261] T 679ea46a35aa4922bd5b199757fbe6bc P 014a92bf31d243febf2877d5869b6138 [term 1 FOLLOWER]: Refusing update from remote peer d45feef75e5b43d0855345b5de047874: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:26:12.804803 27475 raft_consensus.cc:1261] T 679ea46a35aa4922bd5b199757fbe6bc P f46de26a8c5f4921a158eca10ad8970e [term 1 FOLLOWER]: Refusing update from remote peer d45feef75e5b43d0855345b5de047874: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:26:12.805598 27979 consensus_queue.cc:1035] T 679ea46a35aa4922bd5b199757fbe6bc P d45feef75e5b43d0855345b5de047874 [LEADER]: Connected to new peer: Peer: permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:26:12.805966 27843 consensus_queue.cc:1035] T 679ea46a35aa4922bd5b199757fbe6bc P d45feef75e5b43d0855345b5de047874 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:26:12.823215 26980 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:26:12.823493 26980 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:26:12.826983 26980 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b274abd910bd4f9d88d4d17b7bf70fdc: Sending DeleteTablet for 3 replicas of tablet 679ea46a35aa4922bd5b199757fbe6bc
I20260810 23:26:12.827960 27663 tablet_service.cc:1514] Processing DeleteTablet for tablet 679ea46a35aa4922bd5b199757fbe6bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:26:12 UTC) from {username='oryx', principal='oryx/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:54782
I20260810 23:26:12.827946 27436 tablet_service.cc:1514] Processing DeleteTablet for tablet 679ea46a35aa4922bd5b199757fbe6bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:26:12 UTC) from {username='oryx', principal='oryx/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:41168
I20260810 23:26:12.828274 27947 tablet_replica.cc:330] T 679ea46a35aa4922bd5b199757fbe6bc P f46de26a8c5f4921a158eca10ad8970e: stopping tablet replica
I20260810 23:26:12.828316 27201 tablet_service.cc:1514] Processing DeleteTablet for tablet 679ea46a35aa4922bd5b199757fbe6bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:26:12 UTC) from {username='oryx', principal='oryx/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:58188
I20260810 23:26:12.828490 27947 raft_consensus.cc:2229] T 679ea46a35aa4922bd5b199757fbe6bc P f46de26a8c5f4921a158eca10ad8970e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:26:12.828619 27945 tablet_replica.cc:330] T 679ea46a35aa4922bd5b199757fbe6bc P 014a92bf31d243febf2877d5869b6138: stopping tablet replica
I20260810 23:26:12.828729 27947 raft_consensus.cc:2258] T 679ea46a35aa4922bd5b199757fbe6bc P f46de26a8c5f4921a158eca10ad8970e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:26:12.828819 27945 raft_consensus.cc:2229] T 679ea46a35aa4922bd5b199757fbe6bc P 014a92bf31d243febf2877d5869b6138 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:26:12.829049 27945 raft_consensus.cc:2258] T 679ea46a35aa4922bd5b199757fbe6bc P 014a92bf31d243febf2877d5869b6138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:26:12.829064 26980 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:26:12.829370 27875 tablet_replica.cc:330] T 679ea46a35aa4922bd5b199757fbe6bc P d45feef75e5b43d0855345b5de047874: stopping tablet replica
I20260810 23:26:12.829550 27875 raft_consensus.cc:2229] T 679ea46a35aa4922bd5b199757fbe6bc P d45feef75e5b43d0855345b5de047874 [term 1 LEADER]: Raft consensus shutting down.
W20260810 23:26:12.829838 26980 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:26:12.830127 27875 raft_consensus.cc:2258] T 679ea46a35aa4922bd5b199757fbe6bc P d45feef75e5b43d0855345b5de047874 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:26:12.831441 27947 ts_tablet_manager.cc:1905] T 679ea46a35aa4922bd5b199757fbe6bc P f46de26a8c5f4921a158eca10ad8970e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:26:12.831526 27945 ts_tablet_manager.cc:1905] T 679ea46a35aa4922bd5b199757fbe6bc P 014a92bf31d243febf2877d5869b6138: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:26:12.831703 27875 ts_tablet_manager.cc:1905] T 679ea46a35aa4922bd5b199757fbe6bc P d45feef75e5b43d0855345b5de047874: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:26:12.836140 27945 ts_tablet_manager.cc:1918] T 679ea46a35aa4922bd5b199757fbe6bc P 014a92bf31d243febf2877d5869b6138: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:26:12.836293 27945 log.cc:1196] T 679ea46a35aa4922bd5b199757fbe6bc P 014a92bf31d243febf2877d5869b6138: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-2/wal/wals/679ea46a35aa4922bd5b199757fbe6bc
I20260810 23:26:12.836702 27945 ts_tablet_manager.cc:1939] T 679ea46a35aa4922bd5b199757fbe6bc P 014a92bf31d243febf2877d5869b6138: Deleting consensus metadata
I20260810 23:26:12.838016 26956 catalog_manager.cc:4841] TS 014a92bf31d243febf2877d5869b6138 (127.14.175.67:40495): tablet 679ea46a35aa4922bd5b199757fbe6bc (table test-table [id=abd0d89c948f46108bc083c158b411c5]) successfully deleted
I20260810 23:26:12.838184 27947 ts_tablet_manager.cc:1918] T 679ea46a35aa4922bd5b199757fbe6bc P f46de26a8c5f4921a158eca10ad8970e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:26:12.838344 27947 log.cc:1196] T 679ea46a35aa4922bd5b199757fbe6bc P f46de26a8c5f4921a158eca10ad8970e: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-1/wal/wals/679ea46a35aa4922bd5b199757fbe6bc
I20260810 23:26:12.838780 27947 ts_tablet_manager.cc:1939] T 679ea46a35aa4922bd5b199757fbe6bc P f46de26a8c5f4921a158eca10ad8970e: Deleting consensus metadata
I20260810 23:26:12.839872 26953 catalog_manager.cc:4841] TS f46de26a8c5f4921a158eca10ad8970e (127.14.175.66:46177): tablet 679ea46a35aa4922bd5b199757fbe6bc (table test-table [id=abd0d89c948f46108bc083c158b411c5]) successfully deleted
I20260810 23:26:12.840782 27875 ts_tablet_manager.cc:1918] T 679ea46a35aa4922bd5b199757fbe6bc P d45feef75e5b43d0855345b5de047874: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:26:12.840911 27875 log.cc:1196] T 679ea46a35aa4922bd5b199757fbe6bc P d45feef75e5b43d0855345b5de047874: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-0/wal/wals/679ea46a35aa4922bd5b199757fbe6bc
I20260810 23:26:12.841379 27875 ts_tablet_manager.cc:1939] T 679ea46a35aa4922bd5b199757fbe6bc P d45feef75e5b43d0855345b5de047874: Deleting consensus metadata
I20260810 23:26:12.842733 26953 catalog_manager.cc:4841] TS d45feef75e5b43d0855345b5de047874 (127.14.175.65:46357): tablet 679ea46a35aa4922bd5b199757fbe6bc (table test-table [id=abd0d89c948f46108bc083c158b411c5]) successfully deleted
I20260810 23:26:12.843422 27663 tablet_service.cc:1467] Processing CreateTablet for tablet 064c489f7e8a42bc961acb17f7013bf3 (DEFAULT_TABLE table=test-table [id=5751e650924547eba1eae35cfdbfb352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:26:12.843618 27436 tablet_service.cc:1467] Processing CreateTablet for tablet 064c489f7e8a42bc961acb17f7013bf3 (DEFAULT_TABLE table=test-table [id=5751e650924547eba1eae35cfdbfb352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:26:12.843870 27663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 064c489f7e8a42bc961acb17f7013bf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:26:12.844126 27436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 064c489f7e8a42bc961acb17f7013bf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:26:12.849282 27837 tablet_bootstrap.cc:492] T 064c489f7e8a42bc961acb17f7013bf3 P 014a92bf31d243febf2877d5869b6138: Bootstrap starting.
I20260810 23:26:12.850281 27201 tablet_service.cc:1467] Processing CreateTablet for tablet 064c489f7e8a42bc961acb17f7013bf3 (DEFAULT_TABLE table=test-table [id=5751e650924547eba1eae35cfdbfb352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:26:12.850487 27836 tablet_bootstrap.cc:492] T 064c489f7e8a42bc961acb17f7013bf3 P f46de26a8c5f4921a158eca10ad8970e: Bootstrap starting.
I20260810 23:26:12.850746 27201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 064c489f7e8a42bc961acb17f7013bf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:26:12.850947 27837 tablet_bootstrap.cc:654] T 064c489f7e8a42bc961acb17f7013bf3 P 014a92bf31d243febf2877d5869b6138: Neither blocks nor log segments found. Creating new log.
I20260810 23:26:12.852138 27836 tablet_bootstrap.cc:654] T 064c489f7e8a42bc961acb17f7013bf3 P f46de26a8c5f4921a158eca10ad8970e: Neither blocks nor log segments found. Creating new log.
I20260810 23:26:12.858122 27831 tablet_bootstrap.cc:492] T 064c489f7e8a42bc961acb17f7013bf3 P d45feef75e5b43d0855345b5de047874: Bootstrap starting.
I20260810 23:26:12.860222 27831 tablet_bootstrap.cc:654] T 064c489f7e8a42bc961acb17f7013bf3 P d45feef75e5b43d0855345b5de047874: Neither blocks nor log segments found. Creating new log.
I20260810 23:26:12.864449 27836 tablet_bootstrap.cc:492] T 064c489f7e8a42bc961acb17f7013bf3 P f46de26a8c5f4921a158eca10ad8970e: No bootstrap required, opened a new log
I20260810 23:26:12.864696 27836 ts_tablet_manager.cc:1397] T 064c489f7e8a42bc961acb17f7013bf3 P f46de26a8c5f4921a158eca10ad8970e: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260810 23:26:12.865639 27836 raft_consensus.cc:348] T 064c489f7e8a42bc961acb17f7013bf3 P f46de26a8c5f4921a158eca10ad8970e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:12.865924 27836 raft_consensus.cc:374] T 064c489f7e8a42bc961acb17f7013bf3 P f46de26a8c5f4921a158eca10ad8970e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:26:12.866027 27836 raft_consensus.cc:729] T 064c489f7e8a42bc961acb17f7013bf3 P f46de26a8c5f4921a158eca10ad8970e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46de26a8c5f4921a158eca10ad8970e, State: Initialized, Role: FOLLOWER
I20260810 23:26:12.866328 27836 consensus_queue.cc:260] T 064c489f7e8a42bc961acb17f7013bf3 P f46de26a8c5f4921a158eca10ad8970e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:12.867029 27836 ts_tablet_manager.cc:1428] T 064c489f7e8a42bc961acb17f7013bf3 P f46de26a8c5f4921a158eca10ad8970e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 23:26:12.870625 27837 tablet_bootstrap.cc:492] T 064c489f7e8a42bc961acb17f7013bf3 P 014a92bf31d243febf2877d5869b6138: No bootstrap required, opened a new log
I20260810 23:26:12.870960 27837 ts_tablet_manager.cc:1397] T 064c489f7e8a42bc961acb17f7013bf3 P 014a92bf31d243febf2877d5869b6138: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.012s
I20260810 23:26:12.871693 27831 tablet_bootstrap.cc:492] T 064c489f7e8a42bc961acb17f7013bf3 P d45feef75e5b43d0855345b5de047874: No bootstrap required, opened a new log
I20260810 23:26:12.871896 27831 ts_tablet_manager.cc:1397] T 064c489f7e8a42bc961acb17f7013bf3 P d45feef75e5b43d0855345b5de047874: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20260810 23:26:12.872076 27837 raft_consensus.cc:348] T 064c489f7e8a42bc961acb17f7013bf3 P 014a92bf31d243febf2877d5869b6138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:12.872434 27837 raft_consensus.cc:374] T 064c489f7e8a42bc961acb17f7013bf3 P 014a92bf31d243febf2877d5869b6138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:26:12.872568 27837 raft_consensus.cc:729] T 064c489f7e8a42bc961acb17f7013bf3 P 014a92bf31d243febf2877d5869b6138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014a92bf31d243febf2877d5869b6138, State: Initialized, Role: FOLLOWER
I20260810 23:26:12.872831 27837 consensus_queue.cc:260] T 064c489f7e8a42bc961acb17f7013bf3 P 014a92bf31d243febf2877d5869b6138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:12.872759 27831 raft_consensus.cc:348] T 064c489f7e8a42bc961acb17f7013bf3 P d45feef75e5b43d0855345b5de047874 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:12.873045 27831 raft_consensus.cc:374] T 064c489f7e8a42bc961acb17f7013bf3 P d45feef75e5b43d0855345b5de047874 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:26:12.873170 27831 raft_consensus.cc:729] T 064c489f7e8a42bc961acb17f7013bf3 P d45feef75e5b43d0855345b5de047874 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d45feef75e5b43d0855345b5de047874, State: Initialized, Role: FOLLOWER
I20260810 23:26:12.873447 27837 ts_tablet_manager.cc:1428] T 064c489f7e8a42bc961acb17f7013bf3 P 014a92bf31d243febf2877d5869b6138: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:26:12.873452 27831 consensus_queue.cc:260] T 064c489f7e8a42bc961acb17f7013bf3 P d45feef75e5b43d0855345b5de047874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:12.874120 27831 ts_tablet_manager.cc:1428] T 064c489f7e8a42bc961acb17f7013bf3 P d45feef75e5b43d0855345b5de047874: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 23:26:12.901139 27844 raft_consensus.cc:482] T 064c489f7e8a42bc961acb17f7013bf3 P 014a92bf31d243febf2877d5869b6138 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:26:12.901400 27844 raft_consensus.cc:504] T 064c489f7e8a42bc961acb17f7013bf3 P 014a92bf31d243febf2877d5869b6138 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:12.902369 27844 leader_election.cc:288] T 064c489f7e8a42bc961acb17f7013bf3 P 014a92bf31d243febf2877d5869b6138 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d45feef75e5b43d0855345b5de047874 (127.14.175.65:46357), f46de26a8c5f4921a158eca10ad8970e (127.14.175.66:46177)
I20260810 23:26:12.902956 27227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "064c489f7e8a42bc961acb17f7013bf3" candidate_uuid: "014a92bf31d243febf2877d5869b6138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d45feef75e5b43d0855345b5de047874" is_pre_election: true
I20260810 23:26:12.903120 27475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "064c489f7e8a42bc961acb17f7013bf3" candidate_uuid: "014a92bf31d243febf2877d5869b6138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46de26a8c5f4921a158eca10ad8970e" is_pre_election: true
I20260810 23:26:12.903288 27227 raft_consensus.cc:2454] T 064c489f7e8a42bc961acb17f7013bf3 P d45feef75e5b43d0855345b5de047874 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 014a92bf31d243febf2877d5869b6138 in term 0.
I20260810 23:26:12.903378 27475 raft_consensus.cc:2454] T 064c489f7e8a42bc961acb17f7013bf3 P f46de26a8c5f4921a158eca10ad8970e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 014a92bf31d243febf2877d5869b6138 in term 0.
I20260810 23:26:12.903872 27591 leader_election.cc:302] T 064c489f7e8a42bc961acb17f7013bf3 P 014a92bf31d243febf2877d5869b6138 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 014a92bf31d243febf2877d5869b6138, f46de26a8c5f4921a158eca10ad8970e; no voters:
I20260810 23:26:12.904181 27844 raft_consensus.cc:2781] T 064c489f7e8a42bc961acb17f7013bf3 P 014a92bf31d243febf2877d5869b6138 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:26:12.904310 27844 raft_consensus.cc:482] T 064c489f7e8a42bc961acb17f7013bf3 P 014a92bf31d243febf2877d5869b6138 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:26:12.904405 27844 raft_consensus.cc:3037] T 064c489f7e8a42bc961acb17f7013bf3 P 014a92bf31d243febf2877d5869b6138 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:26:12.907507 27844 raft_consensus.cc:504] T 064c489f7e8a42bc961acb17f7013bf3 P 014a92bf31d243febf2877d5869b6138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:12.908239 27844 leader_election.cc:288] T 064c489f7e8a42bc961acb17f7013bf3 P 014a92bf31d243febf2877d5869b6138 [CANDIDATE]: Term 1 election: Requested vote from peers d45feef75e5b43d0855345b5de047874 (127.14.175.65:46357), f46de26a8c5f4921a158eca10ad8970e (127.14.175.66:46177)
I20260810 23:26:12.908838 27475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "064c489f7e8a42bc961acb17f7013bf3" candidate_uuid: "014a92bf31d243febf2877d5869b6138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46de26a8c5f4921a158eca10ad8970e"
I20260810 23:26:12.908856 27227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "064c489f7e8a42bc961acb17f7013bf3" candidate_uuid: "014a92bf31d243febf2877d5869b6138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d45feef75e5b43d0855345b5de047874"
I20260810 23:26:12.909052 27475 raft_consensus.cc:3037] T 064c489f7e8a42bc961acb17f7013bf3 P f46de26a8c5f4921a158eca10ad8970e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:26:12.909090 27227 raft_consensus.cc:3037] T 064c489f7e8a42bc961acb17f7013bf3 P d45feef75e5b43d0855345b5de047874 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:26:12.911882 27475 raft_consensus.cc:2454] T 064c489f7e8a42bc961acb17f7013bf3 P f46de26a8c5f4921a158eca10ad8970e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 014a92bf31d243febf2877d5869b6138 in term 1.
I20260810 23:26:12.912340 27591 leader_election.cc:302] T 064c489f7e8a42bc961acb17f7013bf3 P 014a92bf31d243febf2877d5869b6138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 014a92bf31d243febf2877d5869b6138, f46de26a8c5f4921a158eca10ad8970e; no voters:
I20260810 23:26:12.912550 27227 raft_consensus.cc:2454] T 064c489f7e8a42bc961acb17f7013bf3 P d45feef75e5b43d0855345b5de047874 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 014a92bf31d243febf2877d5869b6138 in term 1.
I20260810 23:26:12.912721 27844 raft_consensus.cc:2781] T 064c489f7e8a42bc961acb17f7013bf3 P 014a92bf31d243febf2877d5869b6138 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:26:12.912882 27844 raft_consensus.cc:686] T 064c489f7e8a42bc961acb17f7013bf3 P 014a92bf31d243febf2877d5869b6138 [term 1 LEADER]: Becoming Leader. State: Replica: 014a92bf31d243febf2877d5869b6138, State: Running, Role: LEADER
I20260810 23:26:12.913273 27844 consensus_queue.cc:237] T 064c489f7e8a42bc961acb17f7013bf3 P 014a92bf31d243febf2877d5869b6138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } }
I20260810 23:26:12.916069 26980 catalog_manager.cc:5495] T 064c489f7e8a42bc961acb17f7013bf3 P 014a92bf31d243febf2877d5869b6138 reported cstate change: term changed from 0 to 1, leader changed from <none> to 014a92bf31d243febf2877d5869b6138 (127.14.175.67). New cstate: current_term: 1 leader_uuid: "014a92bf31d243febf2877d5869b6138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "014a92bf31d243febf2877d5869b6138" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40495 } health_report { overall_health: HEALTHY } } }
I20260810 23:26:12.951202 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:12.944368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54790 (local address 127.14.175.67:40495)
0810 23:26:12.944548 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:12.944554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:12.944573 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:26:12.944773 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:12.944779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:12.944864 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:12.944948 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:26:12.945494 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.946514 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.948526 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:12.949199 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:12.949333 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:12.950039 (+ 706us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:26:12.950226 (+ 187us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:26:12.950316 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:12.950819 (+ 503us) server_negotiation.cc:299] Negotiation successful
0810 23:26:12.950908 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 23:26:12.965504 27475 raft_consensus.cc:1261] T 064c489f7e8a42bc961acb17f7013bf3 P f46de26a8c5f4921a158eca10ad8970e [term 1 FOLLOWER]: Refusing update from remote peer 014a92bf31d243febf2877d5869b6138: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:26:12.966248 27227 raft_consensus.cc:1261] T 064c489f7e8a42bc961acb17f7013bf3 P d45feef75e5b43d0855345b5de047874 [term 1 FOLLOWER]: Refusing update from remote peer 014a92bf31d243febf2877d5869b6138: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:26:12.967133 27844 consensus_queue.cc:1035] T 064c489f7e8a42bc961acb17f7013bf3 P 014a92bf31d243febf2877d5869b6138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46de26a8c5f4921a158eca10ad8970e" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 46177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:26:12.967705 27944 consensus_queue.cc:1035] T 064c489f7e8a42bc961acb17f7013bf3 P 014a92bf31d243febf2877d5869b6138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d45feef75e5b43d0855345b5de047874" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 46357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:26:12.993423 27854 mvcc.cc:205] Tried to move back new op lower bound from 7317112311658373120 to 7317112311455236096. Current Snapshot: MvccSnapshot[applied={T|T < 7317112311658373120}]
I20260810 23:26:13.044484 26980 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:26:13.044798 26980 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:26:13.048281 26980 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b274abd910bd4f9d88d4d17b7bf70fdc: Sending DeleteTablet for 3 replicas of tablet 064c489f7e8a42bc961acb17f7013bf3
I20260810 23:26:13.049225 27663 tablet_service.cc:1514] Processing DeleteTablet for tablet 064c489f7e8a42bc961acb17f7013bf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:26:13 UTC) from {username='oryx', principal='oryx/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:54782
I20260810 23:26:13.049319 27201 tablet_service.cc:1514] Processing DeleteTablet for tablet 064c489f7e8a42bc961acb17f7013bf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:26:13 UTC) from {username='oryx', principal='oryx/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:58188
I20260810 23:26:13.049571 27945 tablet_replica.cc:330] T 064c489f7e8a42bc961acb17f7013bf3 P 014a92bf31d243febf2877d5869b6138: stopping tablet replica
I20260810 23:26:13.049575 27436 tablet_service.cc:1514] Processing DeleteTablet for tablet 064c489f7e8a42bc961acb17f7013bf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:26:13 UTC) from {username='oryx', principal='oryx/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:41168
I20260810 23:26:13.049669 27875 tablet_replica.cc:330] T 064c489f7e8a42bc961acb17f7013bf3 P d45feef75e5b43d0855345b5de047874: stopping tablet replica
I20260810 23:26:13.049772 27945 raft_consensus.cc:2229] T 064c489f7e8a42bc961acb17f7013bf3 P 014a92bf31d243febf2877d5869b6138 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:26:13.049849 27947 tablet_replica.cc:330] T 064c489f7e8a42bc961acb17f7013bf3 P f46de26a8c5f4921a158eca10ad8970e: stopping tablet replica
I20260810 23:26:13.049882 27875 raft_consensus.cc:2229] T 064c489f7e8a42bc961acb17f7013bf3 P d45feef75e5b43d0855345b5de047874 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:26:13.050024 27947 raft_consensus.cc:2229] T 064c489f7e8a42bc961acb17f7013bf3 P f46de26a8c5f4921a158eca10ad8970e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:26:13.050105 27875 raft_consensus.cc:2258] T 064c489f7e8a42bc961acb17f7013bf3 P d45feef75e5b43d0855345b5de047874 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:26:13.050138 27945 raft_consensus.cc:2258] T 064c489f7e8a42bc961acb17f7013bf3 P 014a92bf31d243febf2877d5869b6138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:26:13.050244 27947 raft_consensus.cc:2258] T 064c489f7e8a42bc961acb17f7013bf3 P f46de26a8c5f4921a158eca10ad8970e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:26:13.051261 27947 ts_tablet_manager.cc:1905] T 064c489f7e8a42bc961acb17f7013bf3 P f46de26a8c5f4921a158eca10ad8970e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:26:13.051379 27875 ts_tablet_manager.cc:1905] T 064c489f7e8a42bc961acb17f7013bf3 P d45feef75e5b43d0855345b5de047874: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:26:13.051440 27945 ts_tablet_manager.cc:1905] T 064c489f7e8a42bc961acb17f7013bf3 P 014a92bf31d243febf2877d5869b6138: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:26:13.058279 27947 ts_tablet_manager.cc:1918] T 064c489f7e8a42bc961acb17f7013bf3 P f46de26a8c5f4921a158eca10ad8970e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:26:13.058279 27945 ts_tablet_manager.cc:1918] T 064c489f7e8a42bc961acb17f7013bf3 P 014a92bf31d243febf2877d5869b6138: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:26:13.058434 27945 log.cc:1196] T 064c489f7e8a42bc961acb17f7013bf3 P 014a92bf31d243febf2877d5869b6138: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-2/wal/wals/064c489f7e8a42bc961acb17f7013bf3
I20260810 23:26:13.058434 27947 log.cc:1196] T 064c489f7e8a42bc961acb17f7013bf3 P f46de26a8c5f4921a158eca10ad8970e: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-1/wal/wals/064c489f7e8a42bc961acb17f7013bf3
I20260810 23:26:13.058888 27945 ts_tablet_manager.cc:1939] T 064c489f7e8a42bc961acb17f7013bf3 P 014a92bf31d243febf2877d5869b6138: Deleting consensus metadata
I20260810 23:26:13.058888 27947 ts_tablet_manager.cc:1939] T 064c489f7e8a42bc961acb17f7013bf3 P f46de26a8c5f4921a158eca10ad8970e: Deleting consensus metadata
I20260810 23:26:13.059679 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 27091
I20260810 23:26:13.060142 26953 catalog_manager.cc:4841] TS f46de26a8c5f4921a158eca10ad8970e (127.14.175.66:46177): tablet 064c489f7e8a42bc961acb17f7013bf3 (table test-table [id=5751e650924547eba1eae35cfdbfb352]) successfully deleted
I20260810 23:26:13.060145 26956 catalog_manager.cc:4841] TS 014a92bf31d243febf2877d5869b6138 (127.14.175.67:40495): tablet 064c489f7e8a42bc961acb17f7013bf3 (table test-table [id=5751e650924547eba1eae35cfdbfb352]) successfully deleted
I20260810 23:26:13.063782 27875 ts_tablet_manager.cc:1918] T 064c489f7e8a42bc961acb17f7013bf3 P d45feef75e5b43d0855345b5de047874: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:26:13.063978 27875 log.cc:1196] T 064c489f7e8a42bc961acb17f7013bf3 P d45feef75e5b43d0855345b5de047874: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404299514076-15037-0/minicluster-data/ts-0/wal/wals/064c489f7e8a42bc961acb17f7013bf3
I20260810 23:26:13.064503 27875 ts_tablet_manager.cc:1939] T 064c489f7e8a42bc961acb17f7013bf3 P d45feef75e5b43d0855345b5de047874: Deleting consensus metadata
I20260810 23:26:13.065831 26953 catalog_manager.cc:4841] TS d45feef75e5b43d0855345b5de047874 (127.14.175.65:46357): tablet 064c489f7e8a42bc961acb17f7013bf3 (table test-table [id=5751e650924547eba1eae35cfdbfb352]) successfully deleted
Aug 10 23:26:13 dist-test-slave-rmhg krb5kdc[26900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.175.65@KRBTEST.COM
I20260810 23:26:13.080452 27983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:13.060930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58206 (local address 127.14.175.65:46357)
0810 23:26:13.061132 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:13.061138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:13.061253 (+ 115us) server_negotiation.cc:407] Connection header received
0810 23:26:13.061572 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:13.061578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:13.061680 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:13.061768 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:13.063674 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:13.064856 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:13.066659 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:13.067353 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:13.067488 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:13.071818 (+ 4330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:13.071841 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:13.071865 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:13.071911 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:13.076427 (+ 4516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:13.077395 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:13.077405 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:13.077409 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:13.077644 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:13.078112 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:13.078118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:13.078122 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:13.078618 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:13.079013 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:13.079546 (+ 533us) server_negotiation.cc:299] Negotiation successful
0810 23:26:13.079931 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 23:26:13.081959 27316 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:26:13.218937 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 27091
I20260810 23:26:13.250648 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 27328
Aug 10 23:26:13 dist-test-slave-rmhg krb5kdc[26900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.175.66@KRBTEST.COM
I20260810 23:26:13.266520 28015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:13.251459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41192 (local address 127.14.175.66:46177)
0810 23:26:13.251979 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:13.251984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:13.251999 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:26:13.252054 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:13.252058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:13.252127 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:13.252229 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:13.253610 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:13.254328 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:13.255626 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:13.256060 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:13.256178 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:13.260292 (+ 4114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:13.260310 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:13.260329 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:13.260367 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:13.263422 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:13.264145 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:13.264156 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:13.264158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:13.264327 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:13.264739 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:13.264743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:13.264745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:13.265145 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:13.265482 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:13.265829 (+ 347us) server_negotiation.cc:299] Negotiation successful
0810 23:26:13.266125 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":266,"threads_started":1}
I20260810 23:26:13.267305 27548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:26:13.392628 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 27328
I20260810 23:26:13.423755 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 27563
Aug 10 23:26:13 dist-test-slave-rmhg krb5kdc[26900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.175.67@KRBTEST.COM
I20260810 23:26:13.439807 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:13.424579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54792 (local address 127.14.175.67:40495)
0810 23:26:13.424799 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:13.424804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:13.424830 (+ 26us) server_negotiation.cc:407] Connection header received
0810 23:26:13.425023 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:13.425027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:13.425115 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:13.425241 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:13.426748 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:13.427480 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:13.428787 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:13.429290 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:13.429393 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:13.433925 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:13.433944 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:13.433964 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:13.434018 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:13.437048 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:13.437841 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:13.437850 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:13.437853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:13.438017 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:13.438335 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:13.438339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:13.438341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:13.438807 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:13.439048 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:13.439282 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 23:26:13.439462 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 23:26:13.440537 27781 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:26:13.566780 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 27563
I20260810 23:26:13.591079 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 26919
I20260810 23:26:13.592381 27043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:26:13.711826 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 26919
2026-08-10T23:26:13Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5969 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:26:13 dist-test-slave-rmhg krb5kdc[28026](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:26:13 dist-test-slave-rmhg krb5kdc[28026](info): set up 2 sockets
Aug 10 23:26:13 dist-test-slave-rmhg krb5kdc[28026](info): commencing operation
krb5kdc: starting...
W20260810 23:26:14.973111 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.207s user 0.003s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:26:15 dist-test-slave-rmhg krb5kdc[28026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:26:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:26:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
I20260810 23:26:15.251144 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:41011
--webserver_interface=127.14.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:42959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:41011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:26:15.605429 28049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:26:15.605785 28049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:26:15.605930 28049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:26:15.613582 28049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:26:15.613734 28049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:26:15.613804 28049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:26:15.613883 28049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:26:15.613945 28049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:26:15.624442 28049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:42959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:41011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:41011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:26:15.626020 28049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:26:15.627468 28049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:26:15.639993 28056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:15.640023 28059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:15.640067 28057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:26:15.641060 28049 server_base.cc:1034] running on GCE node
I20260810 23:26:15.641909 28049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:26:15.643380 28049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:26:15.644579 28049 hybrid_clock.cc:648] HybridClock initialized: now 1786404375644517 us; error 53 us; skew 500 ppm
I20260810 23:26:15.644929 28049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T23:26:15Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2003 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:26:15 dist-test-slave-rmhg krb5kdc[28071](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:26:15 dist-test-slave-rmhg krb5kdc[28071](info): set up 2 sockets
Aug 10 23:26:15 dist-test-slave-rmhg krb5kdc[28071](info): commencing operation
krb5kdc: starting...
W20260810 23:26:16.962234 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.192s user 0.003s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:26:17 dist-test-slave-rmhg krb5kdc[28071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:26:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:26:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
I20260810 23:26:17.250502 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:43885
--webserver_interface=127.14.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:41249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:43885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:26:17.604074 28098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:26:17.604436 28098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:26:17.604612 28098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:26:17.612298 28098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:26:17.612465 28098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:26:17.612553 28098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:26:17.612640 28098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:26:17.612705 28098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:26:17.623196 28098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:41249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:43885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:43885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:26:17.624691 28098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:26:17.626153 28098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:26:17.638741 28104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:17.638823 28105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:17.638790 28110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:26:17.639725 28098 server_base.cc:1034] running on GCE node
I20260810 23:26:17.640635 28098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:26:17.642221 28098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:26:17.643436 28098 hybrid_clock.cc:648] HybridClock initialized: now 1786404377643383 us; error 40 us; skew 500 ppm
I20260810 23:26:17.643788 28098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:26:17 dist-test-slave-rmhg krb5kdc[28071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404377, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:26:17.647094 28098 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:26:17.648815 28098 webserver.cc:460] Webserver started at http://127.14.175.126:36785/ using document root <none> and password file <none>
I20260810 23:26:17.649583 28098 fs_manager.cc:359] Metadata directory not provided
I20260810 23:26:17.649711 28098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:26:17.650032 28098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:26:17.653316 28098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "f205060ffea549d3b5a92262c5cd630e"
format_stamp: "Formatted at 2026-08-10 23:26:17 on dist-test-slave-rmhg"
server_key: "31e9013e4cbd8bf36bf856efd1e1ddcd"
server_key_iv: "7f1892c111fcd1df5294e0d7f9397ab2"
server_key_version: "encryptionkey@0"
I20260810 23:26:17.654151 28098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "f205060ffea549d3b5a92262c5cd630e"
format_stamp: "Formatted at 2026-08-10 23:26:17 on dist-test-slave-rmhg"
server_key: "31e9013e4cbd8bf36bf856efd1e1ddcd"
server_key_iv: "7f1892c111fcd1df5294e0d7f9397ab2"
server_key_version: "encryptionkey@0"
I20260810 23:26:17.659824 28098 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260810 23:26:17.663844 28123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:17.665642 28098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:26:17.665858 28098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "f205060ffea549d3b5a92262c5cd630e"
format_stamp: "Formatted at 2026-08-10 23:26:17 on dist-test-slave-rmhg"
server_key: "31e9013e4cbd8bf36bf856efd1e1ddcd"
server_key_iv: "7f1892c111fcd1df5294e0d7f9397ab2"
server_key_version: "encryptionkey@0"
I20260810 23:26:17.666080 28098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:26:17.678819 28098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:26:17.683665 28098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:26:17.683995 28098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:26:17.703365 28098 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:43885
I20260810 23:26:17.703398 28218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:43885 every 8 connection(s)
I20260810 23:26:17.704981 28098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:26:17.708988 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 28098
I20260810 23:26:17.709215 28219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:26:17.709257 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:26:17.709887 15037 external_mini_cluster.cc:1428] Setting key 1bc32b146697a1d941d27cc5fbcbf7e7
I20260810 23:26:17.724659 28219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e: Bootstrap starting.
Aug 10 23:26:17 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:26:17.728893 28219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e: Neither blocks nor log segments found. Creating new log.
I20260810 23:26:17.730437 28219 log.cc:824] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:26:17.734376 28219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e: No bootstrap required, opened a new log
I20260810 23:26:17.735812 28222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:17.711952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38456 (local address 127.14.175.126:43885)
0810 23:26:17.712994 (+ 1042us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:17.713012 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:17.713068 (+ 56us) server_negotiation.cc:407] Connection header received
0810 23:26:17.714784 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:17.714818 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:17.715409 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:17.715929 (+ 520us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:17.717425 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:17.719416 (+ 1991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:17.720834 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:17.722047 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:17.722279 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:17.726433 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:17.726446 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:17.726470 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:17.726549 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:17.730588 (+ 4039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:17.731572 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:17.731581 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:17.731588 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:17.731799 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:17.732315 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:17.732321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:17.732324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:17.733073 (+ 749us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:17.733424 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:17.733886 (+ 462us) server_negotiation.cc:299] Negotiation successful
0810 23:26:17.734372 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":766,"thread_start_us":450,"threads_started":1}
I20260810 23:26:17.743438 28219 raft_consensus.cc:348] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f205060ffea549d3b5a92262c5cd630e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43885 } }
I20260810 23:26:17.743865 28219 raft_consensus.cc:374] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:26:17.743948 28219 raft_consensus.cc:729] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f205060ffea549d3b5a92262c5cd630e, State: Initialized, Role: FOLLOWER
I20260810 23:26:17.744542 28219 consensus_queue.cc:260] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f205060ffea549d3b5a92262c5cd630e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43885 } }
I20260810 23:26:17.744772 28219 raft_consensus.cc:388] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:26:17.744874 28219 raft_consensus.cc:482] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:26:17.744997 28219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:26:17.747267 28219 raft_consensus.cc:504] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f205060ffea549d3b5a92262c5cd630e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43885 } }
I20260810 23:26:17.747792 28219 leader_election.cc:302] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f205060ffea549d3b5a92262c5cd630e; no voters:
I20260810 23:26:17.748394 28219 leader_election.cc:288] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:26:17.748531 28225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:26:17.749645 28225 raft_consensus.cc:686] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [term 1 LEADER]: Becoming Leader. State: Replica: f205060ffea549d3b5a92262c5cd630e, State: Running, Role: LEADER
I20260810 23:26:17.750285 28225 consensus_queue.cc:237] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f205060ffea549d3b5a92262c5cd630e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43885 } }
I20260810 23:26:17.750877 28219 sys_catalog.cc:564] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:26:17.753069 28226 sys_catalog.cc:455] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f205060ffea549d3b5a92262c5cd630e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f205060ffea549d3b5a92262c5cd630e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43885 } } }
I20260810 23:26:17.753582 28226 sys_catalog.cc:458] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [sys.catalog]: This master's current role is: LEADER
I20260810 23:26:17.753096 28228 sys_catalog.cc:455] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f205060ffea549d3b5a92262c5cd630e. Latest consensus state: current_term: 1 leader_uuid: "f205060ffea549d3b5a92262c5cd630e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f205060ffea549d3b5a92262c5cd630e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43885 } } }
I20260810 23:26:17.754441 28228 sys_catalog.cc:458] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [sys.catalog]: This master's current role is: LEADER
I20260810 23:26:17.754501 28235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:26:17.761668 28235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:26:17.771481 28235 catalog_manager.cc:1347] Generated new cluster ID: 00671cb53da240179fe868b3de545a8b
I20260810 23:26:17.771623 28235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:26:17.795032 28235 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:26:17.796581 28235 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:26:17.809212 28235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e: Generated new TSK 0
I20260810 23:26:17.810209 28235 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
I20260810 23:26:17.881781 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.65:0
--local_ip_for_outbound_sockets=127.14.175.65
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:43885
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:41249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:26:18.230517 28259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:26:18.230835 28259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:26:18.230933 28259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:26:18.238457 28259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:26:18.238582 28259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:26:18.238708 28259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.65
I20260810 23:26:18.248859 28259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:41249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:43885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.65
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:26:18.250267 28259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:26:18.251682 28259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:26:18.265853 28268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:18.265847 28270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:18.265908 28267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:26:18.267041 28259 server_base.cc:1034] running on GCE node
I20260810 23:26:18.267738 28259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:26:18.268785 28259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:26:18.270023 28259 hybrid_clock.cc:648] HybridClock initialized: now 1786404378269960 us; error 47 us; skew 500 ppm
I20260810 23:26:18.270398 28259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:26:18 dist-test-slave-rmhg krb5kdc[28071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404378, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:26:18.273763 28259 init.cc:377] Logged in from keytab as kudu/127.14.175.65@KRBTEST.COM (short username kudu)
I20260810 23:26:18.275460 28259 webserver.cc:460] Webserver started at http://127.14.175.65:45237/ using document root <none> and password file <none>
I20260810 23:26:18.276245 28259 fs_manager.cc:359] Metadata directory not provided
I20260810 23:26:18.276382 28259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:26:18.276719 28259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:26:18.279948 28259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-0/data/instance:
uuid: "bbe91efa4bd44bd98d9b4a3faae56f17"
format_stamp: "Formatted at 2026-08-10 23:26:18 on dist-test-slave-rmhg"
server_key: "60d312e48377f097cc0ab4ca7ae4179f"
server_key_iv: "725afd77141e8f189210cd9209bd82d1"
server_key_version: "encryptionkey@0"
I20260810 23:26:18.280846 28259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance:
uuid: "bbe91efa4bd44bd98d9b4a3faae56f17"
format_stamp: "Formatted at 2026-08-10 23:26:18 on dist-test-slave-rmhg"
server_key: "60d312e48377f097cc0ab4ca7ae4179f"
server_key_iv: "725afd77141e8f189210cd9209bd82d1"
server_key_version: "encryptionkey@0"
I20260810 23:26:18.287730 28259 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 23:26:18.291767 28285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:18.293550 28259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:26:18.293743 28259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-0/wal
uuid: "bbe91efa4bd44bd98d9b4a3faae56f17"
format_stamp: "Formatted at 2026-08-10 23:26:18 on dist-test-slave-rmhg"
server_key: "60d312e48377f097cc0ab4ca7ae4179f"
server_key_iv: "725afd77141e8f189210cd9209bd82d1"
server_key_version: "encryptionkey@0"
I20260810 23:26:18.293955 28259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:26:18.311529 28259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:26:18.316470 28259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:26:18.316809 28259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:26:18.317977 28259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:26:18.319519 28259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:26:18.319603 28259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:18.319725 28259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:26:18.319788 28259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:18.352077 28259 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.65:45351
I20260810 23:26:18.352119 28470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.65:45351 every 8 connection(s)
I20260810 23:26:18.353502 28259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
I20260810 23:26:18.359910 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 28259
I20260810 23:26:18.360167 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance
I20260810 23:26:18.360977 15037 external_mini_cluster.cc:1428] Setting key 4af938cea95ddabde6209ee050ce3db5
Aug 10 23:26:18 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404378, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:26:18.376361 28475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:18.356143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:57849 (local address 127.14.175.126:43885)
0810 23:26:18.356587 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:18.356592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:18.357695 (+ 1103us) server_negotiation.cc:407] Connection header received
0810 23:26:18.359560 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:18.359565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:18.359641 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:18.359745 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:18.361958 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:18.362688 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:18.364969 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:18.365449 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:18.365557 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:18.369757 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:18.369762 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:18.369766 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:18.369801 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:18.372537 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:18.373398 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:18.373404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:18.373407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:18.373546 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:18.374048 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:18.374051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:18.374053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:18.374337 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:18.374446 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:18.375468 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0810 23:26:18.375851 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":225,"threads_started":1}
I20260810 23:26:18.377372 28478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:18.356681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.65:57849)
0810 23:26:18.357485 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:18.357529 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:18.359263 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:18.360048 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:18.360061 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:18.360837 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:18.361752 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:18.361774 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:18.362877 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:18.362881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:18.364796 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:18.364813 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:18.365720 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:18.365729 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:18.366057 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:18.366783 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:18.366812 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:18.369506 (+ 2694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:18.372708 (+ 3202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:18.372718 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:18.372737 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:18.373246 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:18.373660 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:18.373665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:18.373668 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:18.373937 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:18.374458 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:18.374467 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:18.374943 (+ 476us) client_negotiation.cc:769] Sending connection context
0810 23:26:18.375276 (+ 333us) client_negotiation.cc:240] Negotiation successful
0810 23:26:18.375851 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":307,"threads_started":1}
I20260810 23:26:18.379158 28473 heartbeater.cc:347] Connected to a master server at 127.14.175.126:43885
I20260810 23:26:18.379505 28473 heartbeater.cc:464] Registering TS with master...
I20260810 23:26:18.380388 28473 heartbeater.cc:511] Master 127.14.175.126:43885 requested a full tablet report, sending...
I20260810 23:26:18.383284 28160 ts_manager.cc:194] Registered new tserver with Master: bbe91efa4bd44bd98d9b4a3faae56f17 (127.14.175.65:45351)
I20260810 23:26:18.384970 28160 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.65@KRBTEST.COM'} at 127.14.175.65:57849
WARNING: no policy specified for kudu/127.14.175.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
I20260810 23:26:18.423692 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.66:0
--local_ip_for_outbound_sockets=127.14.175.66
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:43885
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:41249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:26:18.772840 28482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:26:18.773172 28482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:26:18.773304 28482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:26:18.780970 28482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:26:18.781126 28482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:26:18.781302 28482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.66
I20260810 23:26:18.791594 28482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:41249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:43885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.66
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:26:18.792966 28482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:26:18.794474 28482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:26:18.808018 28494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:18.808090 28492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:18.808082 28498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:26:18.809031 28482 server_base.cc:1034] running on GCE node
I20260810 23:26:18.809801 28482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:26:18.810830 28482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:26:18.812052 28482 hybrid_clock.cc:648] HybridClock initialized: now 1786404378811979 us; error 63 us; skew 500 ppm
I20260810 23:26:18.812408 28482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:26:18 dist-test-slave-rmhg krb5kdc[28071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404378, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:26:18.815728 28482 init.cc:377] Logged in from keytab as kudu/127.14.175.66@KRBTEST.COM (short username kudu)
I20260810 23:26:18.817333 28482 webserver.cc:460] Webserver started at http://127.14.175.66:42795/ using document root <none> and password file <none>
I20260810 23:26:18.818035 28482 fs_manager.cc:359] Metadata directory not provided
I20260810 23:26:18.818147 28482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:26:18.818427 28482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:26:18.821614 28482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-1/data/instance:
uuid: "681b12608374497c800525bbf1f81ce8"
format_stamp: "Formatted at 2026-08-10 23:26:18 on dist-test-slave-rmhg"
server_key: "5bbdbb855a009461c300037ee42b595b"
server_key_iv: "aede90ab3c098f94ae640d84cac342a8"
server_key_version: "encryptionkey@0"
I20260810 23:26:18.822436 28482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance:
uuid: "681b12608374497c800525bbf1f81ce8"
format_stamp: "Formatted at 2026-08-10 23:26:18 on dist-test-slave-rmhg"
server_key: "5bbdbb855a009461c300037ee42b595b"
server_key_iv: "aede90ab3c098f94ae640d84cac342a8"
server_key_version: "encryptionkey@0"
I20260810 23:26:18.827917 28482 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260810 23:26:18.831660 28512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:18.833248 28482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 23:26:18.833426 28482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-1/wal
uuid: "681b12608374497c800525bbf1f81ce8"
format_stamp: "Formatted at 2026-08-10 23:26:18 on dist-test-slave-rmhg"
server_key: "5bbdbb855a009461c300037ee42b595b"
server_key_iv: "aede90ab3c098f94ae640d84cac342a8"
server_key_version: "encryptionkey@0"
I20260810 23:26:18.833635 28482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:26:18.855731 28482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:26:18.860546 28482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:26:18.860864 28482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:26:18.861955 28482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:26:18.863466 28482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:26:18.863554 28482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:18.863683 28482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:26:18.863739 28482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:18.895905 28482 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.66:45669
I20260810 23:26:18.895943 28699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.66:45669 every 8 connection(s)
I20260810 23:26:18.897328 28482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
I20260810 23:26:18.901269 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 28482
I20260810 23:26:18.901467 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance
I20260810 23:26:18.901897 15037 external_mini_cluster.cc:1428] Setting key 719791af702abe4be92a2954ce017371
Aug 10 23:26:18 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404378, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:26:18.919899 28706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:18.900204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:57025 (local address 127.14.175.126:43885)
0810 23:26:18.900647 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:18.900653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:18.902010 (+ 1357us) server_negotiation.cc:407] Connection header received
0810 23:26:18.904380 (+ 2370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:18.904386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:18.904486 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:18.904612 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:18.906356 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:18.907358 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:18.909023 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:18.909673 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:18.909804 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:18.913686 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:18.913693 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:18.913698 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:18.913742 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:18.916469 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:18.917430 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:18.917435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:18.917437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:18.917579 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:18.918070 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:18.918074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:18.918075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:18.918346 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:18.918458 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:18.919383 (+ 925us) server_negotiation.cc:299] Negotiation successful
0810 23:26:18.919582 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":225,"threads_started":1}
I20260810 23:26:18.921432 28707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:18.900822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.66:57025)
0810 23:26:18.901745 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:18.901812 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:18.904096 (+ 2284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:18.904879 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:18.904889 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:18.905539 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:18.906164 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:18.906184 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:18.907506 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:18.907510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:18.908851 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:18.908865 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:18.909929 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:18.909935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:18.910178 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:18.910891 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:18.910921 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:18.913396 (+ 2475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:18.916638 (+ 3242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:18.916647 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:18.916667 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:18.917310 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:18.917706 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:18.917711 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:18.917714 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:18.917976 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:18.918469 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:18.918477 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:18.918947 (+ 470us) client_negotiation.cc:769] Sending connection context
0810 23:26:18.919312 (+ 365us) client_negotiation.cc:240] Negotiation successful
0810 23:26:18.919910 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":340,"threads_started":1}
I20260810 23:26:18.923039 28700 heartbeater.cc:347] Connected to a master server at 127.14.175.126:43885
I20260810 23:26:18.923388 28700 heartbeater.cc:464] Registering TS with master...
I20260810 23:26:18.924360 28700 heartbeater.cc:511] Master 127.14.175.126:43885 requested a full tablet report, sending...
I20260810 23:26:18.926097 28160 ts_manager.cc:194] Registered new tserver with Master: 681b12608374497c800525bbf1f81ce8 (127.14.175.66:45669)
I20260810 23:26:18.926944 28160 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.66@KRBTEST.COM'} at 127.14.175.66:57025
WARNING: no policy specified for kudu/127.14.175.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
I20260810 23:26:18.964421 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.67:0
--local_ip_for_outbound_sockets=127.14.175.67
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:43885
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:41249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:26:19.315668 28713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:26:19.316002 28713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:26:19.316136 28713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:26:19.323686 28713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:26:19.323843 28713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:26:19.324003 28713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.67
I20260810 23:26:19.338768 28713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:41249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:43885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.67
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:26:19.341774 28713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:26:19.344802 28713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:26:19.374812 28720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:19.374715 28726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:19.374850 28722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:26:19.379725 28713 server_base.cc:1034] running on GCE node
I20260810 23:26:19.381016 28713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:26:19.383215 28713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:26:19.384721 28713 hybrid_clock.cc:648] HybridClock initialized: now 1786404379384631 us; error 88 us; skew 500 ppm
I20260810 23:26:19.385412 28713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:26:19.389123 28473 heartbeater.cc:503] Master 127.14.175.126:43885 was elected leader, sending a full tablet report...
Aug 10 23:26:19 dist-test-slave-rmhg krb5kdc[28071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404379, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:26:19.392129 28713 init.cc:377] Logged in from keytab as kudu/127.14.175.67@KRBTEST.COM (short username kudu)
I20260810 23:26:19.395309 28713 webserver.cc:460] Webserver started at http://127.14.175.67:46839/ using document root <none> and password file <none>
I20260810 23:26:19.396672 28713 fs_manager.cc:359] Metadata directory not provided
I20260810 23:26:19.396875 28713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:26:19.397541 28713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:26:19.403484 28713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-2/data/instance:
uuid: "531bc3495a6042d2a6827a7049d39b67"
format_stamp: "Formatted at 2026-08-10 23:26:19 on dist-test-slave-rmhg"
server_key: "c91be0167fb2ee8acc69237f4bb00adc"
server_key_iv: "759ab88a4a942bc71a448fe64e7320be"
server_key_version: "encryptionkey@0"
I20260810 23:26:19.405059 28713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance:
uuid: "531bc3495a6042d2a6827a7049d39b67"
format_stamp: "Formatted at 2026-08-10 23:26:19 on dist-test-slave-rmhg"
server_key: "c91be0167fb2ee8acc69237f4bb00adc"
server_key_iv: "759ab88a4a942bc71a448fe64e7320be"
server_key_version: "encryptionkey@0"
I20260810 23:26:19.415638 28713 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.007s sys 0.001s
I20260810 23:26:19.422917 28737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:19.425666 28713 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260810 23:26:19.425941 28713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-2/wal
uuid: "531bc3495a6042d2a6827a7049d39b67"
format_stamp: "Formatted at 2026-08-10 23:26:19 on dist-test-slave-rmhg"
server_key: "c91be0167fb2ee8acc69237f4bb00adc"
server_key_iv: "759ab88a4a942bc71a448fe64e7320be"
server_key_version: "encryptionkey@0"
I20260810 23:26:19.426259 28713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:26:19.464936 28713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:26:19.475160 28713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:26:19.475750 28713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:26:19.478119 28713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:26:19.482735 28713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:26:19.482872 28713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:19.483078 28713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:26:19.483208 28713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:19.627803 28713 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.67:36341
I20260810 23:26:19.627890 28908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.67:36341 every 8 connection(s)
I20260810 23:26:19.630445 28713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
I20260810 23:26:19.635236 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 28713
I20260810 23:26:19.635614 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance
I20260810 23:26:19.636613 15037 external_mini_cluster.cc:1428] Setting key e331ca3c5598c4a0e6430955619a20f6
Aug 10 23:26:19 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404379, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:26:19.667843 28912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:19.635294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:45913 (local address 127.14.175.126:43885)
0810 23:26:19.636443 (+ 1149us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:19.636452 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:19.638626 (+ 2174us) server_negotiation.cc:407] Connection header received
0810 23:26:19.642093 (+ 3467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:19.642100 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:19.642292 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:19.642458 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:19.645191 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:19.646439 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:19.649337 (+ 2898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:19.650139 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:19.650377 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:19.657304 (+ 6927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:19.657309 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:19.657313 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:19.657378 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:19.661518 (+ 4140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:19.662856 (+ 1338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:19.662876 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:19.662881 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:19.663109 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:19.664128 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:19.664134 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:19.664136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:19.664725 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:19.665019 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:19.666467 (+ 1448us) server_negotiation.cc:299] Negotiation successful
0810 23:26:19.667004 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":899,"thread_start_us":580,"threads_started":1}
I20260810 23:26:19.670212 28914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:19.636613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.67:45913)
0810 23:26:19.638249 (+ 1636us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:19.638410 (+ 161us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:19.641633 (+ 3223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:19.642800 (+ 1167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:19.642813 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:19.643790 (+ 977us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:19.644883 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:19.644916 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:19.646696 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:19.646700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:19.649025 (+ 2325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:19.649052 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:19.650580 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:19.650588 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:19.651192 (+ 604us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:19.652428 (+ 1236us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:19.652512 (+ 84us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:19.656984 (+ 4472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:19.661775 (+ 4791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:19.661793 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:19.661825 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:19.662597 (+ 772us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:19.663292 (+ 695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:19.663313 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:19.663347 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:19.663834 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:19.664942 (+ 1108us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:19.664964 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:19.665698 (+ 734us) client_negotiation.cc:769] Sending connection context
0810 23:26:19.666144 (+ 446us) client_negotiation.cc:240] Negotiation successful
0810 23:26:19.667376 (+ 1232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1081,"thread_start_us":646,"threads_started":1}
I20260810 23:26:19.672571 28910 heartbeater.cc:347] Connected to a master server at 127.14.175.126:43885
I20260810 23:26:19.673033 28910 heartbeater.cc:464] Registering TS with master...
I20260810 23:26:19.674381 28910 heartbeater.cc:511] Master 127.14.175.126:43885 requested a full tablet report, sending...
I20260810 23:26:19.676848 28160 ts_manager.cc:194] Registered new tserver with Master: 531bc3495a6042d2a6827a7049d39b67 (127.14.175.67:36341)
I20260810 23:26:19.678366 28160 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.67@KRBTEST.COM'} at 127.14.175.67:45913
I20260810 23:26:19.684448 15037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.14.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.65.keytab.
Entry for principal oryx/127.14.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.65.keytab.
WARNING: no policy specified for oryx/127.14.175.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.175.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.66.keytab.
Entry for principal oryx/127.14.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.66.keytab.
WARNING: no policy specified for oryx/127.14.175.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.175.67@KRBTEST.COM" created.
I20260810 23:26:19.930598 28700 heartbeater.cc:503] Master 127.14.175.126:43885 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.67.keytab.
Entry for principal oryx/127.14.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.67.keytab.
I20260810 23:26:19.937255 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 28259
Aug 10 23:26:19 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.65@KRBTEST.COM
I20260810 23:26:19.967607 28928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:19.939115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48466 (local address 127.14.175.65:45351)
0810 23:26:19.940259 (+ 1144us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:19.940276 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:19.940331 (+ 55us) server_negotiation.cc:407] Connection header received
0810 23:26:19.940429 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:19.940437 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:19.940804 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:19.941105 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:19.943682 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:19.945521 (+ 1839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:19.947811 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:19.948925 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:19.949129 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:19.956698 (+ 7569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:19.956729 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:19.956762 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:19.956833 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:19.962540 (+ 5707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:19.963979 (+ 1439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:19.963996 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:19.963999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:19.964255 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:19.964883 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:19.964889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:19.964891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:19.965645 (+ 754us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:19.966227 (+ 582us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:19.966544 (+ 317us) server_negotiation.cc:299] Negotiation successful
0810 23:26:19.966916 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":883,"thread_start_us":569,"threads_started":1}
I20260810 23:26:19.970088 28462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:26:20.091431 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 28259
I20260810 23:26:20.129034 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 28482
Aug 10 23:26:20 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.66@KRBTEST.COM
I20260810 23:26:20.160125 28930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:20.130934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35828 (local address 127.14.175.66:45669)
0810 23:26:20.132053 (+ 1119us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:20.132071 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:20.132120 (+ 49us) server_negotiation.cc:407] Connection header received
0810 23:26:20.132206 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:20.132213 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:20.132580 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:20.132798 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:20.135510 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:20.137361 (+ 1851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:20.139986 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:20.141055 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:20.141563 (+ 508us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:20.148601 (+ 7038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:20.148632 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:20.148655 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:20.148731 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:20.154394 (+ 5663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:20.155832 (+ 1438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:20.155851 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:20.155854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:20.156212 (+ 358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:20.157064 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:20.157070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:20.157072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:20.157911 (+ 839us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:20.158371 (+ 460us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:20.158985 (+ 614us) server_negotiation.cc:299] Negotiation successful
0810 23:26:20.159425 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":847,"thread_start_us":574,"threads_started":1}
I20260810 23:26:20.162585 28692 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:26:20.286198 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 28482
I20260810 23:26:20.325613 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 28713
Aug 10 23:26:20 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.67@KRBTEST.COM
I20260810 23:26:20.358980 28932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:20.327313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48354 (local address 127.14.175.67:36341)
0810 23:26:20.328669 (+ 1356us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:20.328680 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:20.328719 (+ 39us) server_negotiation.cc:407] Connection header received
0810 23:26:20.328828 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:20.328836 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:20.329387 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:20.329616 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:20.332574 (+ 2958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:20.334654 (+ 2080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:20.337579 (+ 2925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:20.338753 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:20.338965 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:20.347496 (+ 8531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:20.347527 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:20.347555 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:20.347651 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:20.353381 (+ 5730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:20.354861 (+ 1480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:20.354873 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:20.354882 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:20.355196 (+ 314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:20.355961 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:20.355966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:20.355969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:20.356607 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:20.357075 (+ 468us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:20.357692 (+ 617us) server_negotiation.cc:299] Negotiation successful
0810 23:26:20.358232 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1084,"thread_start_us":769,"threads_started":1}
I20260810 23:26:20.361467 28903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:26:20.492667 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 28713
I20260810 23:26:20.527612 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 28098
I20260810 23:26:20.529631 28212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:26:20.690490 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 28098
I20260810 23:26:20.727057 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:43885
--webserver_interface=127.14.175.126
--webserver_port=36785
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:41249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:43885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:26:21.198659 28936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:26:21.199018 28936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:26:21.199122 28936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:26:21.206622 28936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:26:21.206765 28936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:26:21.206815 28936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:26:21.206874 28936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:26:21.206919 28936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:26:21.217252 28936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:41249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:43885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:43885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=36785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:26:21.218669 28936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:26:21.220112 28936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:26:21.233103 28944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:21.233193 28946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:21.233136 28943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:26:21.233915 28936 server_base.cc:1034] running on GCE node
I20260810 23:26:21.234808 28936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:26:21.236476 28936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:26:21.237737 28936 hybrid_clock.cc:648] HybridClock initialized: now 1786404381237681 us; error 47 us; skew 500 ppm
I20260810 23:26:21.238168 28936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:26:21 dist-test-slave-rmhg krb5kdc[28071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404381, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:26:21.246632 28936 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:26:21.248458 28936 webserver.cc:460] Webserver started at http://127.14.175.126:36785/ using document root <none> and password file <none>
I20260810 23:26:21.249325 28936 fs_manager.cc:359] Metadata directory not provided
I20260810 23:26:21.249482 28936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:26:21.255867 28936 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:26:21.259400 28954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:21.261210 28936 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 23:26:21.261425 28936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "f205060ffea549d3b5a92262c5cd630e"
format_stamp: "Formatted at 2026-08-10 23:26:17 on dist-test-slave-rmhg"
server_key: "31e9013e4cbd8bf36bf856efd1e1ddcd"
server_key_iv: "7f1892c111fcd1df5294e0d7f9397ab2"
server_key_version: "encryptionkey@0"
I20260810 23:26:21.262573 28936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:26:21.277482 28936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:26:21.282406 28936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:26:21.282703 28936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:26:21.302742 28936 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:43885
I20260810 23:26:21.302781 29044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:43885 every 8 connection(s)
I20260810 23:26:21.304371 28936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:26:21.311882 29045 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:26:21.313360 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 28936
I20260810 23:26:21.314397 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.65:45351
--local_ip_for_outbound_sockets=127.14.175.65
--tserver_master_addrs=127.14.175.126:43885
--webserver_port=45237
--webserver_interface=127.14.175.65
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:41249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.175.65
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:26:21.317381 29045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e: Bootstrap starting.
I20260810 23:26:21.340103 29045 log.cc:824] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:26:21.350795 29045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:26:21.351408 29045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e: Bootstrap complete.
I20260810 23:26:21.360996 29045 raft_consensus.cc:348] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f205060ffea549d3b5a92262c5cd630e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43885 } }
I20260810 23:26:21.361968 29045 raft_consensus.cc:729] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f205060ffea549d3b5a92262c5cd630e, State: Initialized, Role: FOLLOWER
I20260810 23:26:21.362730 29045 consensus_queue.cc:260] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f205060ffea549d3b5a92262c5cd630e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43885 } }
I20260810 23:26:21.362982 29045 raft_consensus.cc:388] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:26:21.363104 29045 raft_consensus.cc:482] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:26:21.363273 29045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:26:21.367678 29045 raft_consensus.cc:504] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f205060ffea549d3b5a92262c5cd630e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43885 } }
I20260810 23:26:21.368294 29045 leader_election.cc:302] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f205060ffea549d3b5a92262c5cd630e; no voters:
I20260810 23:26:21.368934 29045 leader_election.cc:288] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 23:26:21.369097 29050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [term 2 FOLLOWER]: Leader election won for term 2
I20260810 23:26:21.371347 29045 sys_catalog.cc:564] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:26:21.376529 29050 raft_consensus.cc:686] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [term 2 LEADER]: Becoming Leader. State: Replica: f205060ffea549d3b5a92262c5cd630e, State: Running, Role: LEADER
I20260810 23:26:21.377434 29050 consensus_queue.cc:237] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f205060ffea549d3b5a92262c5cd630e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43885 } }
I20260810 23:26:21.380450 29051 sys_catalog.cc:455] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f205060ffea549d3b5a92262c5cd630e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f205060ffea549d3b5a92262c5cd630e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43885 } } }
I20260810 23:26:21.380937 29051 sys_catalog.cc:458] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [sys.catalog]: This master's current role is: LEADER
I20260810 23:26:21.380462 29053 sys_catalog.cc:455] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f205060ffea549d3b5a92262c5cd630e. Latest consensus state: current_term: 2 leader_uuid: "f205060ffea549d3b5a92262c5cd630e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f205060ffea549d3b5a92262c5cd630e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43885 } } }
I20260810 23:26:21.381129 29053 sys_catalog.cc:458] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e [sys.catalog]: This master's current role is: LEADER
I20260810 23:26:21.394403 29064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:26:21.400748 29064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:26:21.402386 29064 catalog_manager.cc:1259] Loaded cluster ID: 00671cb53da240179fe868b3de545a8b
I20260810 23:26:21.402508 29064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:26:21.406670 29064 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:26:21.408456 29064 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f205060ffea549d3b5a92262c5cd630e: Loaded TSK: 0
I20260810 23:26:21.410142 29064 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 23:26:21.680552 29047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:26:21.680886 29047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:26:21.681022 29047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:26:21.688618 29047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:26:21.688763 29047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:26:21.688922 29047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.65
I20260810 23:26:21.699254 29047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:41249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.65.keytab
--principal=oryx/127.14.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.65:45351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.175.65
--webserver_port=45237
--tserver_master_addrs=127.14.175.126:43885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.65
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:26:21.700672 29047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:26:21.702143 29047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:26:21.718108 29086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:21.718681 29088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:21.718725 29085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:26:21.718988 29047 server_base.cc:1034] running on GCE node
I20260810 23:26:21.719825 29047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:26:21.721187 29047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:26:21.722468 29047 hybrid_clock.cc:648] HybridClock initialized: now 1786404381722386 us; error 68 us; skew 500 ppm
I20260810 23:26:21.722878 29047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:26:21 dist-test-slave-rmhg krb5kdc[28071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404381, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:26:21.726765 29047 init.cc:377] Logged in from keytab as oryx/127.14.175.65@KRBTEST.COM (short username oryx)
I20260810 23:26:21.728545 29047 webserver.cc:460] Webserver started at http://127.14.175.65:45237/ using document root <none> and password file <none>
I20260810 23:26:21.729352 29047 fs_manager.cc:359] Metadata directory not provided
I20260810 23:26:21.729522 29047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:26:21.736022 29047 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260810 23:26:21.739305 29097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:21.741073 29047 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 23:26:21.741305 29047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-0/wal
uuid: "bbe91efa4bd44bd98d9b4a3faae56f17"
format_stamp: "Formatted at 2026-08-10 23:26:18 on dist-test-slave-rmhg"
server_key: "60d312e48377f097cc0ab4ca7ae4179f"
server_key_iv: "725afd77141e8f189210cd9209bd82d1"
server_key_version: "encryptionkey@0"
I20260810 23:26:21.742377 29047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:26:21.764598 29047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:26:21.770287 29047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:26:21.770716 29047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:26:21.771950 29047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:26:21.773582 29047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:26:21.773710 29047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:21.773833 29047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:26:21.773904 29047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:21.806830 29047 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.65:45351
I20260810 23:26:21.806903 29267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.65:45351 every 8 connection(s)
I20260810 23:26:21.808360 29047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
I20260810 23:26:21.816509 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 29047
I20260810 23:26:21.817317 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.66:45669
--local_ip_for_outbound_sockets=127.14.175.66
--tserver_master_addrs=127.14.175.126:43885
--webserver_port=42795
--webserver_interface=127.14.175.66
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:41249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.175.66
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:26:21 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:21 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:21.828923 29271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:21.811500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.65:56227)
0810 23:26:21.812321 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:21.812371 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:21.814126 (+ 1755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:21.815528 (+ 1402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:21.815536 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:21.816220 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:21.817045 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:21.817073 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:21.818841 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:21.818847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:21.820692 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:21.820711 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:21.821960 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:21.821968 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:21.822249 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:21.823021 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:21.823059 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:21.827690 (+ 4631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":331,"threads_started":1}
W20260810 23:26:21.829489 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:21.811533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:56227 (local address 127.14.175.126:43885)
0810 23:26:21.812288 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:21.812306 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:21.812498 (+ 192us) server_negotiation.cc:407] Connection header received
0810 23:26:21.814677 (+ 2179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:21.814691 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:21.815026 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:21.815315 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:21.817282 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:21.818653 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:21.820860 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:21.821692 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:21.821841 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:21.828185 (+ 6344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.65:56227: BlockingRecv error: recv got EOF from 127.14.175.65:56227 (error 108)
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":288,"threads_started":1}
W20260810 23:26:21.829926 29268 heartbeater.cc:644] Failed to heartbeat to 127.14.175.126:43885 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.175.126:43885: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Aug 10 23:26:21 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:21 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:21.839463 29271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:21.830535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.65:42973)
0810 23:26:21.830744 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:21.830764 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:21.830878 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:21.831167 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:21.831172 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:21.831585 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:21.831821 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:21.831836 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:21.832911 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:21.832916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:21.834236 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:21.834252 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:21.834944 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:21.834949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:21.835116 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:21.835734 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:21.835756 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:21.839150 (+ 3394us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260810 23:26:21.840106 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:21.830736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:42973 (local address 127.14.175.126:43885)
0810 23:26:21.830889 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:21.830894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:21.830912 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:26:21.830970 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:21.830974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:21.831034 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:21.831119 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:21.831973 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:21.832748 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:21.834372 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:21.834785 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:21.834882 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:21.839423 (+ 4541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.65:42973: BlockingRecv error: recv got EOF from 127.14.175.65:42973 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 10 23:26:21 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:21 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:21.849529 29271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:21.840296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.65:37459)
0810 23:26:21.840519 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:21.840541 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:21.840659 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:21.841266 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:21.841270 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:21.841618 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:21.841788 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:21.841798 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:21.843230 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:21.843234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:21.844219 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:21.844232 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:21.845243 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:21.845249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:21.845383 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:21.846093 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:21.846113 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:21.849295 (+ 3182us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":78}
W20260810 23:26:21.849853 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:21.840677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:37459 (local address 127.14.175.126:43885)
0810 23:26:21.840888 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:21.840895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:21.840915 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:26:21.840980 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:21.840985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:21.841081 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:21.841213 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:21.841942 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:21.843070 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:21.844373 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:21.845044 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:21.845226 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:21.849576 (+ 4350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.65:37459: BlockingRecv error: recv got EOF from 127.14.175.65:37459 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260810 23:26:21.849895 29268 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 23:26:22.167999 29274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:26:22.168358 29274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:26:22.168510 29274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:26:22.175935 29274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:26:22.176090 29274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:26:22.176257 29274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.66
I20260810 23:26:22.186321 29274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:41249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.66.keytab
--principal=oryx/127.14.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.66:45669
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.175.66
--webserver_port=42795
--tserver_master_addrs=127.14.175.126:43885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.66
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:26:22.187793 29274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:26:22.189222 29274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:26:22.203195 29283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:22.203267 29286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:22.203231 29281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:26:22.204172 29274 server_base.cc:1034] running on GCE node
I20260810 23:26:22.204849 29274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:26:22.205969 29274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:26:22.207258 29274 hybrid_clock.cc:648] HybridClock initialized: now 1786404382207197 us; error 53 us; skew 500 ppm
I20260810 23:26:22.207629 29274 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:26:22 dist-test-slave-rmhg krb5kdc[28071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404382, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:26:22.210983 29274 init.cc:377] Logged in from keytab as oryx/127.14.175.66@KRBTEST.COM (short username oryx)
I20260810 23:26:22.212577 29274 webserver.cc:460] Webserver started at http://127.14.175.66:42795/ using document root <none> and password file <none>
I20260810 23:26:22.213357 29274 fs_manager.cc:359] Metadata directory not provided
I20260810 23:26:22.213497 29274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:26:22.219084 29274 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:26:22.221951 29295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:22.223425 29274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:26:22.223598 29274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-1/wal
uuid: "681b12608374497c800525bbf1f81ce8"
format_stamp: "Formatted at 2026-08-10 23:26:18 on dist-test-slave-rmhg"
server_key: "5bbdbb855a009461c300037ee42b595b"
server_key_iv: "aede90ab3c098f94ae640d84cac342a8"
server_key_version: "encryptionkey@0"
I20260810 23:26:22.224619 29274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:26:22.251399 29274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:26:22.255839 29274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:26:22.256126 29274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:26:22.257184 29274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:26:22.258564 29274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:26:22.258661 29274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:22.258767 29274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:26:22.258845 29274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:22.290732 29274 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.66:45669
I20260810 23:26:22.290838 29460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.66:45669 every 8 connection(s)
I20260810 23:26:22.292603 29274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
I20260810 23:26:22.300093 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 29274
I20260810 23:26:22.300880 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.67:36341
--local_ip_for_outbound_sockets=127.14.175.67
--tserver_master_addrs=127.14.175.126:43885
--webserver_port=46839
--webserver_interface=127.14.175.67
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:41249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.175.67
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:26:22 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:22 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 23:26:22.311977 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:22.295119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:51215 (local address 127.14.175.126:43885)
0810 23:26:22.295343 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:22.295349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:22.296645 (+ 1296us) server_negotiation.cc:407] Connection header received
0810 23:26:22.298538 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:22.298542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:22.298658 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:22.298749 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:22.300803 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:22.301917 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:22.304512 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:22.305236 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:22.305373 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:22.311725 (+ 6352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.66:51215: BlockingRecv error: recv got EOF from 127.14.175.66:51215 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 23:26:22.312582 29465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:22.295699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.66:51215)
0810 23:26:22.296463 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:22.296502 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:22.298265 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:22.298987 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:22.298996 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:22.299620 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:22.300530 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:22.300561 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:22.302262 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:22.302268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:22.304283 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:22.304302 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:22.305482 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:22.305490 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:22.305796 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:22.306644 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:22.306674 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:22.311487 (+ 4813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":310,"threads_started":1}
W20260810 23:26:22.313288 29462 heartbeater.cc:644] Failed to heartbeat to 127.14.175.126:43885 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.175.126:43885: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Aug 10 23:26:22 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:22 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:22.322559 29465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:22.313878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.66:59011)
0810 23:26:22.314147 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:22.314166 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:22.314286 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:22.314599 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:22.314603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:22.315002 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:22.315179 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:22.315190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:22.316194 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:22.316199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:22.317508 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:22.317524 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:22.318233 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:22.318241 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:22.318430 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:22.318982 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:22.319003 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:22.322294 (+ 3291us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138}
W20260810 23:26:22.322751 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:22.313966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:59011 (local address 127.14.175.126:43885)
0810 23:26:22.314129 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:22.314135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:22.314240 (+ 105us) server_negotiation.cc:407] Connection header received
0810 23:26:22.314409 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:22.314413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:22.314475 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:22.314555 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:22.315325 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:22.316046 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:22.317652 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:22.318060 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:22.318173 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:22.322531 (+ 4358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.66:59011: BlockingRecv error: recv got EOF from 127.14.175.66:59011 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 23:26:22 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:22 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:22.331198 29465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:22.323341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.66:47447)
0810 23:26:22.323500 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:22.323518 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:22.323627 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:22.323955 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:22.323960 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:22.324277 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:22.324431 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:22.324441 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:22.325554 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:22.325558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:22.326476 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:22.326487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:22.327165 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:22.327171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:22.327329 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:22.327867 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:22.327886 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:22.330887 (+ 3001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 23:26:22.331491 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:22.323497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:47447 (local address 127.14.175.126:43885)
0810 23:26:22.323693 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:22.323699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:22.323717 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:26:22.323769 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:22.323773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:22.323830 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:22.323907 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:22.324590 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:22.325420 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:22.326617 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:22.327015 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:22.327125 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:22.331150 (+ 4025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.66:47447: BlockingRecv error: recv got EOF from 127.14.175.66:47447 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260810 23:26:22.331631 29462 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 23:26:22.652347 29467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:26:22.652725 29467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:26:22.652834 29467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:26:22.660357 29467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:26:22.660494 29467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:26:22.660614 29467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.67
I20260810 23:26:22.670949 29467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:41249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/krb5kdc/oryx_127.14.175.67.keytab
--principal=oryx/127.14.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.67:36341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.175.67
--webserver_port=46839
--tserver_master_addrs=127.14.175.126:43885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.67
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:26:22.672432 29467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:26:22.673905 29467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:26:22.687628 29480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:22.687714 29475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:22.687682 29477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:26:22.688895 29467 server_base.cc:1034] running on GCE node
I20260810 23:26:22.689633 29467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:26:22.690682 29467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:26:22.691883 29467 hybrid_clock.cc:648] HybridClock initialized: now 1786404382691832 us; error 47 us; skew 500 ppm
I20260810 23:26:22.692248 29467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:26:22 dist-test-slave-rmhg krb5kdc[28071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404382, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:26:22.695662 29467 init.cc:377] Logged in from keytab as oryx/127.14.175.67@KRBTEST.COM (short username oryx)
I20260810 23:26:22.697125 29467 webserver.cc:460] Webserver started at http://127.14.175.67:46839/ using document root <none> and password file <none>
I20260810 23:26:22.697881 29467 fs_manager.cc:359] Metadata directory not provided
I20260810 23:26:22.698001 29467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:26:22.703751 29467 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:26:22.706743 29490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:22.708268 29467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 23:26:22.708454 29467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-2/wal
uuid: "531bc3495a6042d2a6827a7049d39b67"
format_stamp: "Formatted at 2026-08-10 23:26:19 on dist-test-slave-rmhg"
server_key: "c91be0167fb2ee8acc69237f4bb00adc"
server_key_iv: "759ab88a4a942bc71a448fe64e7320be"
server_key_version: "encryptionkey@0"
I20260810 23:26:22.709576 29467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:26:22.738762 29467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:26:22.743503 29467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:26:22.743793 29467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:26:22.744989 29467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:26:22.746536 29467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:26:22.746649 29467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:22.746759 29467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:26:22.746829 29467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:22.777901 29467 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.67:36341
I20260810 23:26:22.777964 29682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.67:36341 every 8 connection(s)
I20260810 23:26:22.779465 29467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
I20260810 23:26:22.785499 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 29467
Aug 10 23:26:22 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:22 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 23:26:22.797413 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:22.782154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:45867 (local address 127.14.175.126:43885)
0810 23:26:22.782418 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:22.782424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:22.783739 (+ 1315us) server_negotiation.cc:407] Connection header received
0810 23:26:22.785945 (+ 2206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:22.785951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:22.786031 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:22.786135 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:22.788227 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:22.788966 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:22.790744 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:22.791185 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:22.791315 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:22.796963 (+ 5648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.67:45867: BlockingRecv error: recv got EOF from 127.14.175.67:45867 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260810 23:26:22.798094 29685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:22.782788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.67:45867)
0810 23:26:22.783552 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:22.783603 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:22.785576 (+ 1973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:22.786400 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:22.786410 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:22.787216 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:22.788035 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:22.788058 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:22.789126 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:22.789130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:22.790569 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:22.790583 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:22.791453 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:22.791461 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:22.791793 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:22.792488 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:22.792513 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:22.796714 (+ 4201us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":295,"threads_started":1}
W20260810 23:26:22.798822 29683 heartbeater.cc:644] Failed to heartbeat to 127.14.175.126:43885 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.175.126:43885: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
I20260810 23:26:22.803275 29688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:22.786696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34410 (local address 127.14.175.126:43885)
0810 23:26:22.787749 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:22.787755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:22.787772 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:26:22.787826 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:22.787831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:22.787899 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:22.788009 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:22.789656 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:22.790517 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:22.791925 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:22.792351 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:22.792458 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:22.795746 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:22.795755 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:22.795776 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:22.795827 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:22.799659 (+ 3832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:22.800591 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:22.800600 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:22.800611 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:22.800796 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:22.801218 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:22.801224 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:22.801228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:22.801743 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:22.802060 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:22.802548 (+ 488us) server_negotiation.cc:299] Negotiation successful
0810 23:26:22.802954 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":934,"thread_start_us":819,"threads_started":1}
Aug 10 23:26:22 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:22 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:22.808228 29685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:22.799468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.67:58029)
0810 23:26:22.799776 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:22.799792 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:22.799893 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:22.800212 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:22.800215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:22.800516 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:22.800666 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:22.800674 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:22.801812 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:22.801816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:22.802717 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:22.802729 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:22.803647 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:22.803652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:22.803791 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:22.804341 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:22.804358 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:22.807944 (+ 3586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260810 23:26:22.808370 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:22.799538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:58029 (local address 127.14.175.126:43885)
0810 23:26:22.799682 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:22.799686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:22.799847 (+ 161us) server_negotiation.cc:407] Connection header received
0810 23:26:22.800007 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:22.800012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:22.800099 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:22.800199 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:22.800807 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:22.801664 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:22.802999 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:22.803474 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:22.803594 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:22.808152 (+ 4558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.67:58029: BlockingRecv error: recv got EOF from 127.14.175.67:58029 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 10 23:26:22 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:22 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:22.817472 29691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:22.808883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.67:41813)
0810 23:26:22.809560 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:22.809577 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:22.809697 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:22.809991 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:22.809994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:22.810364 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:22.810584 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:22.810592 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:22.811924 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:22.811928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:22.812888 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:22.812900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:22.813694 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:22.813699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:22.813822 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:22.814412 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:22.814430 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:22.817229 (+ 2799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":317,"threads_started":1}
W20260810 23:26:22.817703 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:22.809000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:41813 (local address 127.14.175.126:43885)
0810 23:26:22.809347 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:22.809353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:22.809658 (+ 305us) server_negotiation.cc:407] Connection header received
0810 23:26:22.809798 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:22.809803 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:22.809877 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:22.809983 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:22.810722 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:22.811774 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:22.813013 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:22.813512 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:22.813613 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:22.817443 (+ 3830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.67:41813: BlockingRecv error: recv got EOF from 127.14.175.67:41813 (error 108)
Metrics: {"server-negotiator.queue_time_us":238}
W20260810 23:26:22.817740 29683 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 23:26:22 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:22 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:22.863142 29692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:22.850510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.65:54819)
0810 23:26:22.850999 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:22.851015 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:22.851138 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:22.851438 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:22.851441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:22.851935 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:22.852145 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:22.852156 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:22.853269 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:22.853273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:22.854294 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:22.854308 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:22.855284 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:22.855290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:22.855469 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:22.859625 (+ 4156us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:22.859660 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:22.862796 (+ 3136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":262,"threads_started":1}
W20260810 23:26:22.863349 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:22.850598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:54819 (local address 127.14.175.126:43885)
0810 23:26:22.850842 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:22.850848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:22.851102 (+ 254us) server_negotiation.cc:407] Connection header received
0810 23:26:22.851236 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:22.851240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:22.851308 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:22.851396 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:22.852296 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:22.853097 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:22.854483 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:22.855062 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:22.855225 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:22.863130 (+ 7905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.65:54819: BlockingRecv error: recv got EOF from 127.14.175.65:54819 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 10 23:26:23 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:23 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:23.341225 29693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:23.332389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.66:51943)
0810 23:26:23.332958 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:23.332975 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:23.333124 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:23.333590 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:23.333593 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:23.334184 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:23.334446 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:23.334463 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:23.335729 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:23.335733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:23.336720 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:23.336733 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:23.337482 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:23.337486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:23.337615 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:23.338246 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:23.338264 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:23.340967 (+ 2703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":311,"threads_started":1}
W20260810 23:26:23.341435 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:23.332578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:51943 (local address 127.14.175.126:43885)
0810 23:26:23.332805 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:23.332811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:23.333068 (+ 257us) server_negotiation.cc:407] Connection header received
0810 23:26:23.333342 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:23.333347 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:23.333419 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:23.333499 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:23.334619 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:23.335573 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:23.336865 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:23.337344 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:23.337442 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:23.341216 (+ 3774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.66:51943: BlockingRecv error: recv got EOF from 127.14.175.66:51943 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 10 23:26:23 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:23 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:23.827033 29694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:23.818423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.67:51161)
0810 23:26:23.818957 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:23.818974 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:23.819095 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:23.819405 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:23.819408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:23.819838 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:23.820027 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:23.820035 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:23.821191 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:23.821197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:23.822171 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:23.822184 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:23.822865 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:23.822869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:23.823003 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:23.823594 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:23.823613 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:23.826781 (+ 3168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":265,"threads_started":1}
W20260810 23:26:23.827402 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:23.818634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:51161 (local address 127.14.175.126:43885)
0810 23:26:23.818880 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:23.818886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:23.819046 (+ 160us) server_negotiation.cc:407] Connection header received
0810 23:26:23.819206 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:23.819209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:23.819275 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:23.819360 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:23.820196 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:23.821028 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:23.822311 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:23.822723 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:23.822822 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:23.827152 (+ 4330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.67:51161: BlockingRecv error: recv got EOF from 127.14.175.67:51161 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 10 23:26:23 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:23 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:23.872753 29696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:23.864169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.65:56135)
0810 23:26:23.864751 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:23.864769 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:23.864889 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:23.865238 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:23.865244 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:23.865744 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:23.865958 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:23.865972 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:23.867074 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:23.867078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:23.868044 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:23.868057 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:23.868708 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:23.868712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:23.868827 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:23.869427 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:23.869446 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:23.872392 (+ 2946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":303,"threads_started":1}
W20260810 23:26:23.872879 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:23.864270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:56135 (local address 127.14.175.126:43885)
0810 23:26:23.864509 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:23.864516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:23.864855 (+ 339us) server_negotiation.cc:407] Connection header received
0810 23:26:23.864978 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:23.864984 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:23.865069 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:23.865186 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:23.866110 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:23.866953 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:23.868159 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:23.868574 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:23.868672 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:23.872621 (+ 3949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.65:56135: BlockingRecv error: recv got EOF from 127.14.175.65:56135 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 10 23:26:24 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:24 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:24.351191 29697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:24.342282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.66:47529)
0810 23:26:24.342853 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:24.342869 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:24.342994 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:24.343330 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:24.343334 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:24.343785 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:24.343967 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:24.343975 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:24.345123 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:24.345128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:24.346174 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:24.346189 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:24.346940 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:24.346945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:24.347109 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:24.347709 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:24.347728 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:24.350873 (+ 3145us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":294,"threads_started":1}
W20260810 23:26:24.351437 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:24.342375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:47529 (local address 127.14.175.126:43885)
0810 23:26:24.342586 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:24.342591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:24.342956 (+ 365us) server_negotiation.cc:407] Connection header received
0810 23:26:24.343100 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:24.343104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:24.343193 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:24.343285 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:24.344114 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:24.344946 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:24.346323 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:24.346791 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:24.346895 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:24.351209 (+ 4314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.66:47529: BlockingRecv error: recv got EOF from 127.14.175.66:47529 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 10 23:26:24 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:24 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:24.836654 29698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:24.828241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.67:49357)
0810 23:26:24.828789 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:24.828805 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:24.828938 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:24.829277 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:24.829282 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:24.829753 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:24.829916 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:24.829925 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:24.830957 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:24.830963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:24.831874 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:24.831887 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:24.832614 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:24.832620 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:24.832782 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:24.833383 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:24.833400 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:24.836397 (+ 2997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":300,"threads_started":1}
W20260810 23:26:24.836887 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:24.828403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:49357 (local address 127.14.175.126:43885)
0810 23:26:24.828622 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:24.828628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:24.828879 (+ 251us) server_negotiation.cc:407] Connection header received
0810 23:26:24.829048 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:24.829053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:24.829118 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:24.829241 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:24.830070 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:24.830819 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:24.832029 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:24.832459 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:24.832573 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:24.836623 (+ 4050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.67:49357: BlockingRecv error: recv got EOF from 127.14.175.67:49357 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 10 23:26:24 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:24 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:24.882289 29699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:24.873819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.65:40889)
0810 23:26:24.874302 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:24.874321 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:24.874473 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:24.874819 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:24.874824 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:24.875241 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:24.875428 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:24.875439 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:24.876522 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:24.876527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:24.877485 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:24.877499 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:24.878201 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:24.878207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:24.878384 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:24.878997 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:24.879016 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:24.881917 (+ 2901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":269,"threads_started":1}
W20260810 23:26:24.882459 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:24.873922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:40889 (local address 127.14.175.126:43885)
0810 23:26:24.874160 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:24.874166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:24.874425 (+ 259us) server_negotiation.cc:407] Connection header received
0810 23:26:24.874604 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:24.874608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:24.874677 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:24.874767 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:24.875596 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:24.876353 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:24.877632 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:24.878052 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:24.878171 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:24.882173 (+ 4002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.65:40889: BlockingRecv error: recv got EOF from 127.14.175.65:40889 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 10 23:26:25 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:25 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:25.360865 29701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:25.352200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.66:54889)
0810 23:26:25.352735 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:25.352753 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:25.352886 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:25.353318 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:25.353322 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:25.353687 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:25.353876 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:25.353885 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:25.355121 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:25.355127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:25.356211 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:25.356224 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:25.356921 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:25.356926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:25.357062 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:25.357719 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:25.357739 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:25.360573 (+ 2834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":303,"threads_started":1}
W20260810 23:26:25.361039 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:25.352363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:54889 (local address 127.14.175.126:43885)
0810 23:26:25.352639 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:25.352646 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:25.352833 (+ 187us) server_negotiation.cc:407] Connection header received
0810 23:26:25.353053 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:25.353058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:25.353129 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:25.353261 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:25.354010 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:25.354992 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:25.356335 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:25.356757 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:25.356883 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:25.360818 (+ 3935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.66:54889: BlockingRecv error: recv got EOF from 127.14.175.66:54889 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 10 23:26:25 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:25 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:25.846241 29702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:25.837684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.67:33151)
0810 23:26:25.838166 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:25.838184 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:25.838309 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:25.838626 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:25.838629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:25.838972 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:25.839163 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:25.839171 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:25.840193 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:25.840197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:25.841133 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:25.841175 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:25.841901 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:25.841906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:25.842035 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:25.842622 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:25.842640 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:25.845894 (+ 3254us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":267,"threads_started":1}
W20260810 23:26:25.846328 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:25.837822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:33151 (local address 127.14.175.126:43885)
0810 23:26:25.837991 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:25.837998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:25.838297 (+ 299us) server_negotiation.cc:407] Connection header received
0810 23:26:25.838423 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:25.838427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:25.838499 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:25.838591 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:25.839295 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:25.840047 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:25.841302 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:25.841745 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:25.841843 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:25.846153 (+ 4310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.67:33151: BlockingRecv error: recv got EOF from 127.14.175.67:33151 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 23:26:25 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:25 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:25.892129 29703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:25.883439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.65:40737)
0810 23:26:25.884033 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:25.884054 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:25.884225 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:25.884584 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:25.884589 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:25.885057 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:25.885269 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:25.885281 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:25.886334 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:25.886340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:25.887331 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:25.887344 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:25.888135 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:25.888141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:25.888289 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:25.888863 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:25.888883 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:25.891822 (+ 2939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":328,"threads_started":1}
W20260810 23:26:25.892321 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:25.883468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:40737 (local address 127.14.175.126:43885)
0810 23:26:25.883726 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:25.883732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:25.884153 (+ 421us) server_negotiation.cc:407] Connection header received
0810 23:26:25.884357 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:25.884363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:25.884431 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:25.884528 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:25.885420 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:25.886195 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:25.887483 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:25.887951 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:25.888083 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:25.892081 (+ 3998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.65:40737: BlockingRecv error: recv got EOF from 127.14.175.65:40737 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 10 23:26:26 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:26 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:26.373554 29705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:26.361845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.66:45123)
0810 23:26:26.362313 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:26.362331 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:26.362459 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:26.362780 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:26.362783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:26.363182 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:26.363344 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:26.363353 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:26.364414 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:26.364419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:26.365604 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:26.365620 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:26.366513 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:26.366522 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:26.366676 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:26.370259 (+ 3583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:26.370280 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:26.373311 (+ 3031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":244,"threads_started":1}
W20260810 23:26:26.373765 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:26.361963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:45123 (local address 127.14.175.126:43885)
0810 23:26:26.362215 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:26.362221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:26.362408 (+ 187us) server_negotiation.cc:407] Connection header received
0810 23:26:26.362560 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:26.362563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:26.362654 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:26.362745 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:26.363468 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:26.364234 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:26.365789 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:26.366252 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:26.366375 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:26.373521 (+ 7146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.66:45123: BlockingRecv error: recv got EOF from 127.14.175.66:45123 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Aug 10 23:26:26 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:26 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:26.855892 29708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:26.847352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.67:60711)
0810 23:26:26.847933 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:26.847950 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:26.848072 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:26.848344 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:26.848347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:26.848766 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:26.848931 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:26.848941 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:26.850050 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:26.850053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:26.851066 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:26.851077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:26.851757 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:26.851761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:26.851885 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:26.852470 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:26.852487 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:26.855560 (+ 3073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":351,"threads_started":1}
W20260810 23:26:26.856081 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:26.847411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:60711 (local address 127.14.175.126:43885)
0810 23:26:26.847581 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:26.847588 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:26.848032 (+ 444us) server_negotiation.cc:407] Connection header received
0810 23:26:26.848165 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:26.848169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:26.848232 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:26.848313 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:26.849078 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:26.849937 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:26.851197 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:26.851642 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:26.851738 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:26.855835 (+ 4097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.67:60711: BlockingRecv error: recv got EOF from 127.14.175.67:60711 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 10 23:26:26 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:26 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:26.902714 29710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:26.893315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.65:40661)
0810 23:26:26.893903 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:26.893920 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:26.894075 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:26.894441 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:26.894447 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:26.894889 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:26.895069 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:26.895079 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:26.896329 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:26.896335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:26.897601 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:26.897619 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:26.898398 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:26.898403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:26.898555 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:26.899212 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:26.899233 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:26.902428 (+ 3195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":260,"threads_started":1}
W20260810 23:26:26.902943 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:26.893536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:40661 (local address 127.14.175.126:43885)
0810 23:26:26.893793 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:26.893800 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:26.894012 (+ 212us) server_negotiation.cc:407] Connection header received
0810 23:26:26.894221 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:26.894225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:26.894295 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:26.894382 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:26.895260 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:26.896124 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:26.897757 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:26.898228 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:26.898336 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:26.902718 (+ 4382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.65:40661: BlockingRecv error: recv got EOF from 127.14.175.65:40661 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 10 23:26:27 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:27 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:27.385196 29712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:27.374584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.66:44453)
0810 23:26:27.375236 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:27.375257 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:27.375404 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:27.375788 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:27.375794 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:27.376359 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:27.376586 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:27.376599 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:27.377767 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:27.377773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:27.379318 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:27.379335 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:27.380107 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:27.380114 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:27.380304 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:27.380993 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:27.381019 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:27.384780 (+ 3761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":334,"threads_started":1}
W20260810 23:26:27.385439 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:27.374747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:44453 (local address 127.14.175.126:43885)
0810 23:26:27.374918 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:27.374925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:27.375344 (+ 419us) server_negotiation.cc:407] Connection header received
0810 23:26:27.375582 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:27.375586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:27.375652 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:27.375725 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:27.376752 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:27.377628 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:27.379504 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:27.379949 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:27.380058 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:27.385132 (+ 5074us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.66:44453: BlockingRecv error: recv got EOF from 127.14.175.66:44453 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 10 23:26:27 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:27 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:27.865666 29714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:27.856999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.67:51151)
0810 23:26:27.857627 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:27.857644 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:27.857773 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:27.858129 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:27.858133 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:27.858566 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:27.858738 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:27.858748 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:27.859740 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:27.859744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:27.860690 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:27.860702 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:27.861645 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:27.861650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:27.861791 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:27.862345 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:27.862362 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:27.865413 (+ 3051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":327,"threads_started":1}
W20260810 23:26:27.865801 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:27.857081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:51151 (local address 127.14.175.126:43885)
0810 23:26:27.857297 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:27.857301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:27.857784 (+ 483us) server_negotiation.cc:407] Connection header received
0810 23:26:27.857878 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:27.857882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:27.857950 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:27.858030 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:27.858868 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:27.859611 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:27.860858 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:27.861451 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:27.861598 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:27.865590 (+ 3992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.67:51151: BlockingRecv error: recv got EOF from 127.14.175.67:51151 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Aug 10 23:26:27 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:27 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:27.912102 29715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:27.903783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.65:58651)
0810 23:26:27.904313 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:27.904330 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:27.904485 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:27.904808 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:27.904812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:27.905331 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:27.905506 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:27.905518 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:27.906558 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:27.906563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:27.907483 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:27.907496 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:27.908207 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:27.908213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:27.908377 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:27.908982 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:27.909001 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:27.911796 (+ 2795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":286,"threads_started":1}
W20260810 23:26:27.912293 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:27.903874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:58651 (local address 127.14.175.126:43885)
0810 23:26:27.904108 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:27.904116 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:27.904423 (+ 307us) server_negotiation.cc:407] Connection header received
0810 23:26:27.904607 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:27.904611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:27.904676 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:27.904770 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:27.905667 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:27.906411 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:27.907625 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:27.908046 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:27.908159 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:27.912024 (+ 3865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.65:58651: BlockingRecv error: recv got EOF from 127.14.175.65:58651 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 10 23:26:28 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:28 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:28.395368 29721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:28.386534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.66:49749)
0810 23:26:28.387069 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:28.387090 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:28.387218 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:28.387536 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:28.387539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:28.387956 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:28.388113 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:28.388122 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:28.389420 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:28.389425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:28.390713 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:28.390727 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:28.391488 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:28.391492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:28.391618 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:28.392210 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:28.392228 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:28.395056 (+ 2828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":291,"threads_started":1}
W20260810 23:26:28.395617 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:28.386660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:49749 (local address 127.14.175.126:43885)
0810 23:26:28.386840 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:28.386845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:28.387173 (+ 328us) server_negotiation.cc:407] Connection header received
0810 23:26:28.387326 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:28.387330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:28.387396 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:28.387484 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:28.388267 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:28.389258 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:28.390865 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:28.391346 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:28.391456 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:28.395332 (+ 3876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.66:49749: BlockingRecv error: recv got EOF from 127.14.175.66:49749 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 10 23:26:28 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:28 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:28.875612 29722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:28.866650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.67:59919)
0810 23:26:28.867324 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:28.867344 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:28.867538 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:28.867848 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:28.867851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:28.868336 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:28.868498 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:28.868507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:28.869594 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:28.869599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:28.870501 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:28.870513 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:28.871289 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:28.871294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:28.871444 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:28.872120 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:28.872140 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:28.875365 (+ 3225us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":418,"threads_started":1}
W20260810 23:26:28.875849 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:28.866769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:59919 (local address 127.14.175.126:43885)
0810 23:26:28.866971 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:28.866977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:28.867416 (+ 439us) server_negotiation.cc:407] Connection header received
0810 23:26:28.867660 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:28.867664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:28.867729 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:28.867807 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:28.868625 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:28.869441 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:28.870642 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:28.871095 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:28.871211 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:28.875593 (+ 4382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.67:59919: BlockingRecv error: recv got EOF from 127.14.175.67:59919 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 10 23:26:28 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:28 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:28.922078 29723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:28.913480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.65:41565)
0810 23:26:28.914136 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:28.914154 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:28.914280 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:28.914564 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:28.914567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:28.915014 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:28.915216 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:28.915226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:28.916380 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:28.916385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:28.917358 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:28.917372 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:28.918127 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:28.918134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:28.918293 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:28.918895 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:28.918914 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:28.921712 (+ 2798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":266,"threads_started":1}
W20260810 23:26:28.922365 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:28.913603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:41565 (local address 127.14.175.126:43885)
0810 23:26:28.913795 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:28.913800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:28.914226 (+ 426us) server_negotiation.cc:407] Connection header received
0810 23:26:28.914372 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:28.914376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:28.914442 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:28.914535 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:28.915385 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:28.916226 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:28.917510 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:28.917955 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:28.918072 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:28.921982 (+ 3910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.65:41565: BlockingRecv error: recv got EOF from 127.14.175.65:41565 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 10 23:26:29 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:29 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:29.405380 29724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:29.396398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.66:33587)
0810 23:26:29.397023 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:29.397040 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:29.397222 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:29.397684 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:29.397689 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:29.398126 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:29.398302 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:29.398313 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:29.399383 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:29.399387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:29.400330 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:29.400342 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:29.401053 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:29.401059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:29.401257 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:29.402010 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:29.402036 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:29.404996 (+ 2960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":281,"threads_started":1}
W20260810 23:26:29.405539 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:29.396502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:33587 (local address 127.14.175.126:43885)
0810 23:26:29.396774 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:29.396782 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:29.397137 (+ 355us) server_negotiation.cc:407] Connection header received
0810 23:26:29.397458 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:29.397463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:29.397531 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:29.397615 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:29.398460 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:29.399238 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:29.400478 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:29.400888 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:29.401011 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:29.405288 (+ 4277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.66:33587: BlockingRecv error: recv got EOF from 127.14.175.66:33587 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
Aug 10 23:26:29 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:29 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:29.885537 29725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:29.876584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.67:49347)
0810 23:26:29.877301 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:29.877321 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:29.877468 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:29.877850 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:29.877860 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:29.878307 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:29.878480 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:29.878490 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:29.879535 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:29.879540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:29.880466 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:29.880478 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:29.881464 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:29.881469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:29.881599 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:29.882161 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:29.882177 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:29.885230 (+ 3053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":293,"threads_started":1}
W20260810 23:26:29.885691 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:29.876743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:49347 (local address 127.14.175.126:43885)
0810 23:26:29.876970 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:29.876976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:29.877414 (+ 438us) server_negotiation.cc:407] Connection header received
0810 23:26:29.877633 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:29.877637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:29.877705 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:29.877789 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:29.878654 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:29.879386 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:29.880638 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:29.881108 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:29.881250 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:29.885408 (+ 4158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.67:49347: BlockingRecv error: recv got EOF from 127.14.175.67:49347 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 10 23:26:29 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:29 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:29.930999 29726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:29.923069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.65:36375)
0810 23:26:29.923657 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:29.923675 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:29.923792 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:29.924074 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:29.924077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:29.924495 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:29.924656 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:29.924665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:29.925658 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:29.925662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:29.926594 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:29.926606 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:29.927268 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:29.927272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:29.927390 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:29.927949 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:29.927966 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:29.930767 (+ 2801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":320,"threads_started":1}
W20260810 23:26:29.931210 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:29.923225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:36375 (local address 127.14.175.126:43885)
0810 23:26:29.923396 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:29.923400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:29.923768 (+ 368us) server_negotiation.cc:407] Connection header received
0810 23:26:29.923894 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:29.923897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:29.923964 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:29.924047 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:29.924771 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:29.925535 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:29.926708 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:29.927123 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:29.927212 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:29.930962 (+ 3750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.65:36375: BlockingRecv error: recv got EOF from 127.14.175.65:36375 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 10 23:26:30 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:30 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:30.414911 29728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:30.406357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.66:47875)
0810 23:26:30.406913 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:30.406930 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:30.407064 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:30.407397 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:30.407402 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:30.407898 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:30.408054 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:30.408064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:30.409341 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:30.409345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:30.410281 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:30.410293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:30.411051 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:30.411058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:30.411206 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:30.411849 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:30.411866 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:30.414683 (+ 2817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":295,"threads_started":1}
W20260810 23:26:30.415212 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:30.406483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:47875 (local address 127.14.175.126:43885)
0810 23:26:30.406685 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:30.406692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:30.407012 (+ 320us) server_negotiation.cc:407] Connection header received
0810 23:26:30.407196 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:30.407200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:30.407264 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:30.407347 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:30.408221 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:30.409194 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:30.410422 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:30.410866 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:30.410982 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:30.414922 (+ 3940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.66:47875: BlockingRecv error: recv got EOF from 127.14.175.66:47875 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 10 23:26:30 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:30 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:30.895252 29729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:30.886558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.67:50445)
0810 23:26:30.887140 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:30.887158 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:30.887282 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:30.887649 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:30.887652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:30.888058 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:30.888222 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:30.888231 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:30.889362 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:30.889366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:30.890515 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:30.890530 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:30.891408 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:30.891415 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:30.891547 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:30.892196 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:30.892215 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:30.895025 (+ 2810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":288,"threads_started":1}
W20260810 23:26:30.895552 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:30.886732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:50445 (local address 127.14.175.126:43885)
0810 23:26:30.886937 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:30.886942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:30.887270 (+ 328us) server_negotiation.cc:407] Connection header received
0810 23:26:30.887387 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:30.887391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:30.887486 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:30.887590 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:30.888378 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:30.889199 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:30.890651 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:30.891218 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:30.891324 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:30.895267 (+ 3943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.67:50445: BlockingRecv error: recv got EOF from 127.14.175.67:50445 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 10 23:26:30 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:30 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:30.940760 29730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:30.931924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.65:45377)
0810 23:26:30.932495 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:30.932512 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:30.932650 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:30.932983 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:30.932988 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:30.933521 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:30.933712 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:30.933723 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:30.934743 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:30.934748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:30.935871 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:30.935886 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:30.936620 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:30.936625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:30.936806 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:30.937417 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:30.937435 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:30.940506 (+ 3071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":304,"threads_started":1}
W20260810 23:26:30.940927 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:30.932066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:45377 (local address 127.14.175.126:43885)
0810 23:26:30.932274 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:30.932281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:30.932594 (+ 313us) server_negotiation.cc:407] Connection header received
0810 23:26:30.932766 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:30.932770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:30.932849 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:30.932933 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:30.933860 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:30.934594 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:30.936018 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:30.936460 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:30.936577 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:30.940730 (+ 4153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.65:45377: BlockingRecv error: recv got EOF from 127.14.175.65:45377 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 10 23:26:31 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:31 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:31.427740 29731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:31.415884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.66:58647)
0810 23:26:31.416458 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:31.416474 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:31.416609 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:31.416979 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:31.416984 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:31.417448 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:31.417619 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:31.417629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:31.418685 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:31.418689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:31.419659 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:31.419672 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:31.420410 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:31.420416 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:31.420583 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:31.424349 (+ 3766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:31.424376 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:31.427494 (+ 3118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":313,"threads_started":1}
W20260810 23:26:31.427937 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:31.416009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:58647 (local address 127.14.175.126:43885)
0810 23:26:31.416275 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:31.416282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:31.416555 (+ 273us) server_negotiation.cc:407] Connection header received
0810 23:26:31.416769 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:31.416775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:31.416842 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:31.416926 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:31.417774 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:31.418542 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:31.419801 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:31.420256 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:31.420380 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:31.427710 (+ 7330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.66:58647: BlockingRecv error: recv got EOF from 127.14.175.66:58647 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 10 23:26:31 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:31 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:31.905001 29732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:31.896325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.67:33823)
0810 23:26:31.896874 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:31.896890 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:31.897022 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:31.897373 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:31.897377 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:31.897826 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:31.898011 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:31.898021 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:31.899234 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:31.899240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:31.900487 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:31.900503 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:31.901252 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:31.901257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:31.901385 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:31.901934 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:31.901951 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:31.904743 (+ 2792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":323,"threads_started":1}
W20260810 23:26:31.905207 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:31.896468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:33823 (local address 127.14.175.126:43885)
0810 23:26:31.896645 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:31.896650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:31.896966 (+ 316us) server_negotiation.cc:407] Connection header received
0810 23:26:31.897125 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:31.897128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:31.897222 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:31.897315 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:31.898190 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:31.899079 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:31.900627 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:31.901080 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:31.901215 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:31.904970 (+ 3755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.67:33823: BlockingRecv error: recv got EOF from 127.14.175.67:33823 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 10 23:26:31 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:31 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:31.950381 29733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:31.941835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.65:36973)
0810 23:26:31.942330 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:31.942348 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:31.942464 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:31.942755 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:31.942759 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:31.943169 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:31.943325 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:31.943335 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:31.944380 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:31.944385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:31.945414 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:31.945428 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:31.946134 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:31.946138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:31.946267 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:31.946909 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:31.946932 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:31.950142 (+ 3210us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":285,"threads_started":1}
W20260810 23:26:31.950567 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:31.941952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:36973 (local address 127.14.175.126:43885)
0810 23:26:31.942125 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:31.942130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:31.942417 (+ 287us) server_negotiation.cc:407] Connection header received
0810 23:26:31.942551 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:31.942555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:31.942634 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:31.942720 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:31.943461 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:31.944241 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:31.945549 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:31.946008 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:31.946104 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:31.950325 (+ 4221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.65:36973: BlockingRecv error: recv got EOF from 127.14.175.65:36973 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 10 23:26:32 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:32 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:32.437553 29735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:32.428688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.66:47475)
0810 23:26:32.429318 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:32.429336 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:32.429455 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:32.429773 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:32.429776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:32.430181 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:32.430339 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:32.430351 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:32.431553 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:32.431557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:32.432876 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:32.432890 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:32.433645 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:32.433651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:32.433772 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:32.434397 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:32.434415 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:32.437253 (+ 2838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":369,"threads_started":1}
W20260810 23:26:32.437691 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:32.428805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:47475 (local address 127.14.175.126:43885)
0810 23:26:32.429017 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:32.429022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:32.429417 (+ 395us) server_negotiation.cc:407] Connection header received
0810 23:26:32.429557 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:32.429561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:32.429651 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:32.429740 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:32.430473 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:32.431405 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:32.432997 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:32.433502 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:32.433628 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:32.437455 (+ 3827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.66:47475: BlockingRecv error: recv got EOF from 127.14.175.66:47475 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 23:26:32 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:32 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:32.914486 29736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:32.906128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.67:43499)
0810 23:26:32.906671 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:32.906688 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:32.906812 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:32.907087 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:32.907090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:32.907477 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:32.907672 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:32.907681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:32.908700 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:32.908704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:32.909676 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:32.909690 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:32.910326 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:32.910329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:32.910445 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:32.911096 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:32.911115 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:32.914167 (+ 3052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":300,"threads_started":1}
W20260810 23:26:32.914702 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:32.906385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:43499 (local address 127.14.175.126:43885)
0810 23:26:32.906591 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:32.906598 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:32.906762 (+ 164us) server_negotiation.cc:407] Connection header received
0810 23:26:32.906902 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:32.906906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:32.906972 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:32.907055 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:32.907798 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:32.908543 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:32.909784 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:32.910203 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:32.910296 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:32.914442 (+ 4146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.67:43499: BlockingRecv error: recv got EOF from 127.14.175.67:43499 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 10 23:26:32 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:32 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:32.959599 29738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:32.951268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.65:56575)
0810 23:26:32.951942 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:32.951962 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:32.952086 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:32.952377 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:32.952380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:32.952830 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:32.953013 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:32.953023 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:32.954076 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:32.954080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:32.955008 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:32.955021 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:32.955690 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:32.955695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:32.955822 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:32.956416 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:32.956435 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:32.959337 (+ 2902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":430,"threads_started":1}
W20260810 23:26:32.959755 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:32.951331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:56575 (local address 127.14.175.126:43885)
0810 23:26:32.951569 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:32.951576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:32.952060 (+ 484us) server_negotiation.cc:407] Connection header received
0810 23:26:32.952183 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:32.952187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:32.952268 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:32.952349 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:32.953197 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:32.953933 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:32.955130 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:32.955556 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:32.955650 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:32.959549 (+ 3899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.65:56575: BlockingRecv error: recv got EOF from 127.14.175.65:56575 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 10 23:26:33 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:33 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:33.446972 29739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:33.438485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.66:55873)
0810 23:26:33.438988 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:33.439004 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:33.439137 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:33.439475 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:33.439480 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:33.439938 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:33.440130 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:33.440140 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:33.441200 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:33.441204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:33.442115 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:33.442127 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:33.442845 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:33.442851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:33.443024 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:33.443670 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:33.443688 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:33.446680 (+ 2992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":261,"threads_started":1}
W20260810 23:26:33.447131 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:33.438612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:55873 (local address 127.14.175.126:43885)
0810 23:26:33.438809 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:33.438816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:33.439087 (+ 271us) server_negotiation.cc:407] Connection header received
0810 23:26:33.439265 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:33.439269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:33.439340 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:33.439421 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:33.440278 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:33.441028 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:33.442259 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:33.442691 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:33.442803 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:33.446892 (+ 4089us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.66:55873: BlockingRecv error: recv got EOF from 127.14.175.66:55873 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 10 23:26:33 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:33 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:33.923642 29740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:33.915480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.67:51667)
0810 23:26:33.915981 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:33.915999 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:33.916121 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:33.916401 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:33.916404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:33.916798 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:33.917000 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:33.917009 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:33.918113 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:33.918118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:33.919086 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:33.919100 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:33.919814 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:33.919819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:33.919946 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:33.920579 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:33.920597 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:33.923394 (+ 2797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":264,"threads_started":1}
W20260810 23:26:33.923831 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:33.915587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:51667 (local address 127.14.175.126:43885)
0810 23:26:33.915799 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:33.915805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:33.916080 (+ 275us) server_negotiation.cc:407] Connection header received
0810 23:26:33.916231 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:33.916235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:33.916299 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:33.916373 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:33.917200 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:33.917967 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:33.919218 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:33.919652 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:33.919749 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:33.923634 (+ 3885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.67:51667: BlockingRecv error: recv got EOF from 127.14.175.67:51667 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 10 23:26:33 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:33 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:33.969988 29744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:33.960822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.65:48801)
0810 23:26:33.961359 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:33.961378 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:33.961570 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:33.961983 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:33.961988 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:33.962518 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:33.962694 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:33.962704 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:33.963930 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:33.963935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:33.964893 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:33.964907 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:33.965686 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:33.965692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:33.965881 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:33.966505 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:33.966524 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:33.969615 (+ 3091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":308,"threads_started":1}
W20260810 23:26:33.970135 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:33.960937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:48801 (local address 127.14.175.126:43885)
0810 23:26:33.961228 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:33.961235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:33.961482 (+ 247us) server_negotiation.cc:407] Connection header received
0810 23:26:33.961706 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:33.961710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:33.961820 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:33.961937 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:33.962839 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:33.963784 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:33.965043 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:33.965543 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:33.965658 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:33.969877 (+ 4219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.65:48801: BlockingRecv error: recv got EOF from 127.14.175.65:48801 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 10 23:26:34 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:34 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:34.457334 29746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:34.448052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.66:41675)
0810 23:26:34.448604 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:34.448625 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:34.448763 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:34.449124 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:34.449127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:34.449575 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:34.449746 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:34.449756 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:34.450975 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:34.450980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:34.452197 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:34.452216 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:34.452937 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:34.452941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:34.453102 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:34.453780 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:34.453807 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:34.456956 (+ 3149us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":320,"threads_started":1}
W20260810 23:26:34.457520 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:34.448126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:41675 (local address 127.14.175.126:43885)
0810 23:26:34.448370 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:34.448375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:34.448714 (+ 339us) server_negotiation.cc:407] Connection header received
0810 23:26:34.448896 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:34.448901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:34.448973 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:34.449069 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:34.449892 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:34.450809 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:34.452355 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:34.452804 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:34.452921 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:34.457288 (+ 4367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.66:41675: BlockingRecv error: recv got EOF from 127.14.175.66:41675 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 10 23:26:34 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:34 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:34.933820 29748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:34.924802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.67:43621)
0810 23:26:34.925420 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:34.925437 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:34.925596 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:34.925939 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:34.925944 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:34.926385 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:34.926549 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:34.926559 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:34.927713 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:34.927717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:34.928781 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:34.928798 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:34.929713 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:34.929719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:34.929914 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:34.930543 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:34.930561 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:34.933587 (+ 3026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":376,"threads_started":1}
W20260810 23:26:34.934084 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:34.924908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:43621 (local address 127.14.175.126:43885)
0810 23:26:34.925208 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:34.925215 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:34.925534 (+ 319us) server_negotiation.cc:407] Connection header received
0810 23:26:34.925731 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:34.925735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:34.925800 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:34.925886 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:34.926767 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:34.927540 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:34.928970 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:34.929538 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:34.929663 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:34.933818 (+ 4155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.67:43621: BlockingRecv error: recv got EOF from 127.14.175.67:43621 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Aug 10 23:26:34 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:34 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:34.979592 29752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:34.971048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.65:43623)
0810 23:26:34.971647 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:34.971663 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:34.971824 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:34.972172 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:34.972177 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:34.972606 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:34.972788 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:34.972802 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:34.973890 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:34.973896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:34.974854 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:34.974866 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:34.975637 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:34.975643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:34.975801 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:34.976389 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:34.976407 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:34.979336 (+ 2929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":295,"threads_started":1}
W20260810 23:26:34.979763 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:34.971154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:43623 (local address 127.14.175.126:43885)
0810 23:26:34.971364 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:34.971369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:34.971754 (+ 385us) server_negotiation.cc:407] Connection header received
0810 23:26:34.971936 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:34.971940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:34.972026 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:34.972108 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:34.972929 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:34.973742 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:34.975002 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:34.975456 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:34.975572 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:34.979542 (+ 3970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.65:43623: BlockingRecv error: recv got EOF from 127.14.175.65:43623 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 10 23:26:35 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:35 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:35.466792 29753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:35.458302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.66:43091)
0810 23:26:35.458856 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:35.458873 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:35.458987 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:35.459270 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:35.459274 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:35.459690 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:35.459857 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:35.459867 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:35.460923 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:35.460927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:35.461931 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:35.461945 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:35.462746 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:35.462751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:35.462875 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:35.463452 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:35.463475 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:35.466417 (+ 2942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":300,"threads_started":1}
W20260810 23:26:35.466907 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:35.458450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:43091 (local address 127.14.175.126:43885)
0810 23:26:35.458607 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:35.458612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:35.458937 (+ 325us) server_negotiation.cc:407] Connection header received
0810 23:26:35.459079 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:35.459082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:35.459158 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:35.459239 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:35.459970 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:35.460787 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:35.462099 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:35.462544 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:35.462658 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:35.466662 (+ 4004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.66:43091: BlockingRecv error: recv got EOF from 127.14.175.66:43091 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 23:26:35 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:35 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:35.943624 29754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:35.934821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.67:50909)
0810 23:26:35.935419 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:35.935435 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:35.935586 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:35.935917 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:35.935922 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:35.936358 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:35.936523 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:35.936533 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:35.937663 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:35.937667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:35.938605 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:35.938616 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:35.939322 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:35.939328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:35.939507 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:35.940197 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:35.940215 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:35.943275 (+ 3060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":332,"threads_started":1}
W20260810 23:26:35.943796 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:35.934924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:50909 (local address 127.14.175.126:43885)
0810 23:26:35.935154 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:35.935161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:35.935552 (+ 391us) server_negotiation.cc:407] Connection header received
0810 23:26:35.935709 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:35.935713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:35.935777 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:35.935860 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:35.936690 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:35.937518 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:35.938754 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:35.939175 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:35.939288 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:35.943532 (+ 4244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.67:50909: BlockingRecv error: recv got EOF from 127.14.175.67:50909 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 10 23:26:35 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:35 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:35.988956 29755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:35.980570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.65:60305)
0810 23:26:35.981283 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:35.981300 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:35.981434 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:35.981780 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:35.981785 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:35.982195 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:35.982384 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:35.982394 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:35.983457 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:35.983463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:35.984395 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:35.984408 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:35.985118 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:35.985124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:35.985331 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:35.985915 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:35.985933 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:35.988708 (+ 2775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":322,"threads_started":1}
W20260810 23:26:35.989264 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:35.980736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:60305 (local address 127.14.175.126:43885)
0810 23:26:35.981117 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:35.981123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:35.981379 (+ 256us) server_negotiation.cc:407] Connection header received
0810 23:26:35.981558 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:35.981561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:35.981638 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:35.981721 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:35.982534 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:35.983307 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:35.984536 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:35.984970 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:35.985085 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:35.988952 (+ 3867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.65:60305: BlockingRecv error: recv got EOF from 127.14.175.65:60305 (error 108)
Metrics: {"server-negotiator.queue_time_us":246}
Aug 10 23:26:36 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:36 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:36.480252 29756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:36.467898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.66:46055)
0810 23:26:36.468398 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:36.468414 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:36.468556 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:36.468897 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:36.468901 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:36.469370 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:36.469560 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:36.469569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:36.470619 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:36.470624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:36.471776 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:36.471792 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:36.472800 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:36.472806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:36.472984 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:36.477093 (+ 4109us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:36.477115 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:36.479992 (+ 2877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":275,"threads_started":1}
W20260810 23:26:36.480449 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:36.467999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:46055 (local address 127.14.175.126:43885)
0810 23:26:36.468204 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:36.468210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:36.468504 (+ 294us) server_negotiation.cc:407] Connection header received
0810 23:26:36.468698 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:36.468702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:36.468767 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:36.468850 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:36.469717 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:36.470461 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:36.471933 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:36.472636 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:36.472778 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:36.480250 (+ 7472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.66:46055: BlockingRecv error: recv got EOF from 127.14.175.66:46055 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 10 23:26:36 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:36 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.67@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:36.953867 29757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:36.944820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.67:51605)
0810 23:26:36.945584 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:36.945605 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:36.945744 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:36.946167 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:36.946171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:36.946625 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:36.946855 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:36.946867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:36.948073 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:36.948078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:36.949068 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:36.949083 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:36.949918 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:36.949923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:36.950076 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:36.950680 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:36.950699 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:36.953617 (+ 2918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":501,"threads_started":1}
W20260810 23:26:36.954144 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:36.944891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:51605 (local address 127.14.175.126:43885)
0810 23:26:36.945218 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:36.945226 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:36.945701 (+ 475us) server_negotiation.cc:407] Connection header received
0810 23:26:36.945851 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:36.945857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:36.945942 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:36.946064 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:36.947044 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:36.947897 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:36.949254 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:36.949754 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:36.949858 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:36.953889 (+ 4031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.67:51605: BlockingRecv error: recv got EOF from 127.14.175.67:51605 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 10 23:26:36 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:36 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.65@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:36.999094 29760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:36.990244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.65:44587)
0810 23:26:36.990811 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:36.990828 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:36.990992 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:36.991327 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:36.991332 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:36.991764 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:36.991943 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:36.991953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:36.993033 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:36.993037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:36.994050 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:36.994065 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:36.995012 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:36.995018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:36.995185 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:36.995797 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:36.995822 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:36.998724 (+ 2902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":301,"threads_started":1}
W20260810 23:26:36.999197 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:36.990373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:44587 (local address 127.14.175.126:43885)
0810 23:26:36.990603 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:36.990610 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:36.990935 (+ 325us) server_negotiation.cc:407] Connection header received
0810 23:26:36.991112 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:36.991116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:36.991184 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:36.991270 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:36.992094 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:36.992881 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:36.994198 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:36.994849 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:36.994971 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:36.998959 (+ 3988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.65:44587: BlockingRecv error: recv got EOF from 127.14.175.65:44587 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 10 23:26:37 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:26:37 dist-test-slave-rmhg krb5kdc[28071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.175.66@KRBTEST.COM for oryx/127.14.175.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:26:37.490345 29761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:37.481544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43885 (local address 127.14.175.66:56633)
0810 23:26:37.482119 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:37.482140 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:37.482289 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:37.482708 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:37.482712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:37.483132 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:37.483316 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:37.483326 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:37.484396 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:37.484400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:37.485395 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:37.485409 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:37.486193 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:37.486198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:37.486328 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:37.486954 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:37.486973 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:37.489929 (+ 2956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.175.126:43885: Server oryx/127.14.175.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":281,"threads_started":1}
W20260810 23:26:37.490494 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:37.481609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:56633 (local address 127.14.175.126:43885)
0810 23:26:37.481853 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:37.481861 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:37.482266 (+ 405us) server_negotiation.cc:407] Connection header received
0810 23:26:37.482405 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:37.482410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:37.482487 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:37.482598 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:37.483464 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:37.484248 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:37.485533 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:37.486017 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:37.486129 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:37.490235 (+ 4106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.175.66:56633: BlockingRecv error: recv got EOF from 127.14.175.66:56633 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
I20260810 23:26:37.790287 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 29047
I20260810 23:26:37.803299 29764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:37.791526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55270 (local address 127.14.175.65:45351)
0810 23:26:37.792207 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:37.792217 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:37.792250 (+ 33us) server_negotiation.cc:407] Connection header received
0810 23:26:37.792449 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:37.792458 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:37.792712 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:37.792877 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:37.794487 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:37.795587 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:37.796826 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:37.797482 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:37.797591 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:37.800114 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:37.800123 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:37.800140 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:37.800177 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:37.802764 (+ 2587us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.14.175.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0810 23:26:37.803058 (+ 294us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55270: Request ticket server kudu/127.14.175.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":338,"threads_started":1}
W20260810 23:26:37.803429 29764 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55270: Request ticket server kudu/127.14.175.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260810 23:26:37.803534 15037 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 29047: Not authorized: Client connection negotiation failed: client connection to 127.14.175.65:45351: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.14.175.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260810 23:26:37.803767 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 29047
I20260810 23:26:37.830948 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 29274
I20260810 23:26:37.843478 29766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:37.831997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54734 (local address 127.14.175.66:45669)
0810 23:26:37.832596 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:37.832604 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:37.832639 (+ 35us) server_negotiation.cc:407] Connection header received
0810 23:26:37.832705 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:37.832711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:37.832942 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:37.833107 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:37.834376 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:37.835536 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:37.836783 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:37.837456 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:37.837571 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:37.840144 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:37.840154 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:37.840171 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:37.840211 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:37.842791 (+ 2580us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.14.175.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0810 23:26:37.843139 (+ 348us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54734: Request ticket server kudu/127.14.175.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":243,"threads_started":1}
W20260810 23:26:37.843603 29766 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54734: Request ticket server kudu/127.14.175.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260810 23:26:37.843647 15037 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 29274: Not authorized: Client connection negotiation failed: client connection to 127.14.175.66:45669: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.14.175.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260810 23:26:37.843798 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 29274
I20260810 23:26:37.868010 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 29467
I20260810 23:26:37.880541 29767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:37.869085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39032 (local address 127.14.175.67:36341)
0810 23:26:37.869765 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:37.869773 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:37.869802 (+ 29us) server_negotiation.cc:407] Connection header received
0810 23:26:37.869866 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:37.869873 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:37.870142 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:37.870301 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:37.871625 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:37.872771 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:37.874084 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:37.874671 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:37.874777 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:37.877374 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:37.877383 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:37.877400 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:37.877437 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:37.879954 (+ 2517us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.14.175.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0810 23:26:37.880276 (+ 322us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39032: Request ticket server kudu/127.14.175.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":381,"threads_started":1}
W20260810 23:26:37.880657 29767 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39032: Request ticket server kudu/127.14.175.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260810 23:26:37.880751 15037 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 29467: Not authorized: Client connection negotiation failed: client connection to 127.14.175.67:36341: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.14.175.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260810 23:26:37.880934 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 29467
I20260810 23:26:37.906006 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 28936
I20260810 23:26:37.907449 29039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:26:38.030547 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 28936
2026-08-10T23:26:38Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (22314 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T23:26:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:26:38Z Disabled control of system clock
I20260810 23:26:38.125360 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:36271
--webserver_interface=127.14.175.126
--webserver_port=0
--builtin_ntp_servers=127.14.175.84:43507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:36271
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 23:26:38.485682 29775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:26:38.485986 29775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:26:38.486068 29775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:26:38.493427 29775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:26:38.493557 29775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:26:38.493605 29775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:26:38.493664 29775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:26:38.493714 29775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:26:38.503192 29775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:43507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:36271
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:36271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:26:38.504601 29775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:26:38.506446 29775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:26:38.518908 29783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:38.518949 29786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:38.518973 29788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:26:38.519707 29775 server_base.cc:1034] running on GCE node
I20260810 23:26:38.520619 29775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:26:38.522568 29775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:26:38.523835 29775 hybrid_clock.cc:648] HybridClock initialized: now 1786404398523770 us; error 57 us; skew 500 ppm
I20260810 23:26:38.524243 29775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:26:38.526852 29775 webserver.cc:460] Webserver started at http://127.14.175.126:36167/ using document root <none> and password file <none>
I20260810 23:26:38.527622 29775 fs_manager.cc:359] Metadata directory not provided
I20260810 23:26:38.527753 29775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:26:38.528120 29775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:26:38.531464 29775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "2b5a3a63b3ac45868e7f567a9b423f60"
format_stamp: "Formatted at 2026-08-10 23:26:38 on dist-test-slave-rmhg"
server_key: "d2cd2c066c07d54c32a3bfee4369d88e"
server_key_iv: "86ba9153c3c96a0c84cda6e34e0e5c7b"
server_key_version: "encryptionkey@0"
I20260810 23:26:38.532389 29775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "2b5a3a63b3ac45868e7f567a9b423f60"
format_stamp: "Formatted at 2026-08-10 23:26:38 on dist-test-slave-rmhg"
server_key: "d2cd2c066c07d54c32a3bfee4369d88e"
server_key_iv: "86ba9153c3c96a0c84cda6e34e0e5c7b"
server_key_version: "encryptionkey@0"
I20260810 23:26:38.538321 29775 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 23:26:38.542215 29799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:38.543803 29775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:26:38.543995 29775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "2b5a3a63b3ac45868e7f567a9b423f60"
format_stamp: "Formatted at 2026-08-10 23:26:38 on dist-test-slave-rmhg"
server_key: "d2cd2c066c07d54c32a3bfee4369d88e"
server_key_iv: "86ba9153c3c96a0c84cda6e34e0e5c7b"
server_key_version: "encryptionkey@0"
I20260810 23:26:38.544220 29775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:26:38.563794 29775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:26:38.564795 29775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:26:38.565099 29775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:26:38.583966 29775 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:36271
I20260810 23:26:38.583985 29866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:36271 every 8 connection(s)
I20260810 23:26:38.585529 29775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:26:38.589879 29867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:26:38.594035 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 29775
I20260810 23:26:38.594260 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:26:38.594950 15037 external_mini_cluster.cc:1428] Setting key f8e7062c462dff66188995c46943f2a4
I20260810 23:26:38.603267 29867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b5a3a63b3ac45868e7f567a9b423f60: Bootstrap starting.
I20260810 23:26:38.607276 29867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b5a3a63b3ac45868e7f567a9b423f60: Neither blocks nor log segments found. Creating new log.
I20260810 23:26:38.608704 29867 log.cc:824] T 00000000000000000000000000000000 P 2b5a3a63b3ac45868e7f567a9b423f60: Log is configured to *not* fsync() on all Append() calls
I20260810 23:26:38.611932 29871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:38.597276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34354 (local address 127.14.175.126:36271)
0810 23:26:38.598477 (+ 1201us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:38.598493 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:38.598551 (+ 58us) server_negotiation.cc:407] Connection header received
0810 23:26:38.601140 (+ 2589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:38.601198 (+ 58us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:38.601886 (+ 688us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:38.602410 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:38.603010 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:38.605013 (+ 2003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:38.606479 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:38.607577 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:38.607812 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:38.608932 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:38.608943 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:38.608964 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:26:38.609770 (+ 806us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:38.609857 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:26:38.609888 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:38.609994 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:38.610251 (+ 257us) server_negotiation.cc:299] Negotiation successful
0810 23:26:38.610618 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":380,"threads_started":1}
I20260810 23:26:38.612550 29867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b5a3a63b3ac45868e7f567a9b423f60: No bootstrap required, opened a new log
I20260810 23:26:38.620817 29867 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b5a3a63b3ac45868e7f567a9b423f60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5a3a63b3ac45868e7f567a9b423f60" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 36271 } }
I20260810 23:26:38.621374 29867 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b5a3a63b3ac45868e7f567a9b423f60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:26:38.621479 29867 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b5a3a63b3ac45868e7f567a9b423f60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b5a3a63b3ac45868e7f567a9b423f60, State: Initialized, Role: FOLLOWER
I20260810 23:26:38.622148 29867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b5a3a63b3ac45868e7f567a9b423f60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5a3a63b3ac45868e7f567a9b423f60" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 36271 } }
I20260810 23:26:38.622388 29867 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b5a3a63b3ac45868e7f567a9b423f60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:26:38.622599 29867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b5a3a63b3ac45868e7f567a9b423f60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:26:38.622808 29867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b5a3a63b3ac45868e7f567a9b423f60 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:26:38.625468 29867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b5a3a63b3ac45868e7f567a9b423f60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5a3a63b3ac45868e7f567a9b423f60" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 36271 } }
I20260810 23:26:38.626004 29867 leader_election.cc:302] T 00000000000000000000000000000000 P 2b5a3a63b3ac45868e7f567a9b423f60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b5a3a63b3ac45868e7f567a9b423f60; no voters:
I20260810 23:26:38.626646 29867 leader_election.cc:288] T 00000000000000000000000000000000 P 2b5a3a63b3ac45868e7f567a9b423f60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:26:38.626796 29876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b5a3a63b3ac45868e7f567a9b423f60 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:26:38.627511 29876 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b5a3a63b3ac45868e7f567a9b423f60 [term 1 LEADER]: Becoming Leader. State: Replica: 2b5a3a63b3ac45868e7f567a9b423f60, State: Running, Role: LEADER
I20260810 23:26:38.628103 29876 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b5a3a63b3ac45868e7f567a9b423f60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5a3a63b3ac45868e7f567a9b423f60" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 36271 } }
I20260810 23:26:38.628788 29867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b5a3a63b3ac45868e7f567a9b423f60 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:26:38.631678 29878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b5a3a63b3ac45868e7f567a9b423f60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b5a3a63b3ac45868e7f567a9b423f60. Latest consensus state: current_term: 1 leader_uuid: "2b5a3a63b3ac45868e7f567a9b423f60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5a3a63b3ac45868e7f567a9b423f60" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 36271 } } }
I20260810 23:26:38.632128 29878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b5a3a63b3ac45868e7f567a9b423f60 [sys.catalog]: This master's current role is: LEADER
I20260810 23:26:38.632681 29877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b5a3a63b3ac45868e7f567a9b423f60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b5a3a63b3ac45868e7f567a9b423f60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5a3a63b3ac45868e7f567a9b423f60" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 36271 } } }
I20260810 23:26:38.633116 29877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b5a3a63b3ac45868e7f567a9b423f60 [sys.catalog]: This master's current role is: LEADER
I20260810 23:26:38.638767 29888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:26:38.644591 29888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:26:38.655381 29888 catalog_manager.cc:1347] Generated new cluster ID: 9420eb94f78243e89d4e02d70b211dac
I20260810 23:26:38.655517 29888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:26:38.672401 29888 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:26:38.673831 29888 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:26:38.686451 29888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b5a3a63b3ac45868e7f567a9b423f60: Generated new TSK 0
I20260810 23:26:38.687402 29888 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:26:38.698961 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.65:0
--local_ip_for_outbound_sockets=127.14.175.65
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:36271
--builtin_ntp_servers=127.14.175.84:43507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 23:26:39.079412 29907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:26:39.079743 29907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:26:39.079869 29907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:26:39.087320 29907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:26:39.087459 29907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:26:39.087618 29907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.65
I20260810 23:26:39.097038 29907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:43507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:36271
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.65
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:26:39.098476 29907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:26:39.100320 29907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:26:39.114243 29917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:39.114554 29913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:39.114895 29914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:26:39.115011 29907 server_base.cc:1034] running on GCE node
I20260810 23:26:39.115710 29907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:26:39.116695 29907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:26:39.117921 29907 hybrid_clock.cc:648] HybridClock initialized: now 1786404399117857 us; error 54 us; skew 500 ppm
I20260810 23:26:39.118268 29907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:26:39.120734 29907 webserver.cc:460] Webserver started at http://127.14.175.65:45555/ using document root <none> and password file <none>
I20260810 23:26:39.121598 29907 fs_manager.cc:359] Metadata directory not provided
I20260810 23:26:39.121721 29907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:26:39.122071 29907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:26:39.125422 29907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/data/instance:
uuid: "c31aa931d24b4152bc46a15e4fdede51"
format_stamp: "Formatted at 2026-08-10 23:26:39 on dist-test-slave-rmhg"
server_key: "1ed9abf142e67daa43c363bdc6cfd4ac"
server_key_iv: "88f85cb75c9faa587bf0b82e626b3170"
server_key_version: "encryptionkey@0"
I20260810 23:26:39.126291 29907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance:
uuid: "c31aa931d24b4152bc46a15e4fdede51"
format_stamp: "Formatted at 2026-08-10 23:26:39 on dist-test-slave-rmhg"
server_key: "1ed9abf142e67daa43c363bdc6cfd4ac"
server_key_iv: "88f85cb75c9faa587bf0b82e626b3170"
server_key_version: "encryptionkey@0"
I20260810 23:26:39.132090 29907 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:26:39.135900 29934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:39.137488 29907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:26:39.137687 29907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/wal
uuid: "c31aa931d24b4152bc46a15e4fdede51"
format_stamp: "Formatted at 2026-08-10 23:26:39 on dist-test-slave-rmhg"
server_key: "1ed9abf142e67daa43c363bdc6cfd4ac"
server_key_iv: "88f85cb75c9faa587bf0b82e626b3170"
server_key_version: "encryptionkey@0"
I20260810 23:26:39.137920 29907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:26:39.156539 29907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:26:39.157838 29907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:26:39.158159 29907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:26:39.159209 29907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:26:39.160781 29907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:26:39.160876 29907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:39.160967 29907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:26:39.161024 29907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:39.192893 29907 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.65:45763
I20260810 23:26:39.192958 30121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.65:45763 every 8 connection(s)
I20260810 23:26:39.194276 29907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
I20260810 23:26:39.202739 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 29907
I20260810 23:26:39.202940 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance
I20260810 23:26:39.203573 15037 external_mini_cluster.cc:1428] Setting key 34f381db68cc578069e94997ece5fe86
I20260810 23:26:39.206555 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.66:0
--local_ip_for_outbound_sockets=127.14.175.66
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:36271
--builtin_ntp_servers=127.14.175.84:43507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 23:26:39.208077 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:39.197415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:36271 (local address 127.14.175.65:53913)
0810 23:26:39.198358 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:39.198395 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:39.200064 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:39.200791 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:39.200800 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:39.200888 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:39.201513 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:39.201531 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:39.202598 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:39.202602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:39.203997 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:39.204016 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:39.204869 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:39.204877 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:39.205225 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:39.206026 (+ 801us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:26:39.206029 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:39.206072 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:26:39.206160 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:39.206532 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:39.206546 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:39.206553 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 23:26:39.206752 (+ 199us) client_negotiation.cc:240] Negotiation successful
0810 23:26:39.207082 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":293,"threads_started":1}
I20260810 23:26:39.208716 30126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:39.196826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:53913 (local address 127.14.175.126:36271)
0810 23:26:39.197299 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:39.197304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:39.198531 (+ 1227us) server_negotiation.cc:407] Connection header received
0810 23:26:39.200344 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:39.200349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:39.200432 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:39.200533 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:39.201687 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:39.202443 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:39.204188 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:39.204634 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:39.204738 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:39.206359 (+ 1621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:39.206366 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:39.206369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:26:39.206386 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:39.206427 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:26:39.206440 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:39.206517 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:39.208339 (+ 1822us) server_negotiation.cc:299] Negotiation successful
0810 23:26:39.208445 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":215,"threads_started":1}
I20260810 23:26:39.209918 30122 heartbeater.cc:347] Connected to a master server at 127.14.175.126:36271
I20260810 23:26:39.210235 30122 heartbeater.cc:464] Registering TS with master...
I20260810 23:26:39.211038 30122 heartbeater.cc:511] Master 127.14.175.126:36271 requested a full tablet report, sending...
I20260810 23:26:39.213197 29824 ts_manager.cc:194] Registered new tserver with Master: c31aa931d24b4152bc46a15e4fdede51 (127.14.175.65:45763)
I20260810 23:26:39.214730 29824 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.175.65:53913
W20260810 23:26:39.557255 30129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:26:39.557555 30129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:26:39.557685 30129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:26:39.565205 30129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:26:39.565325 30129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:26:39.565472 30129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.66
I20260810 23:26:39.574882 30129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:43507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:36271
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.66
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:26:39.576237 30129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:26:39.578121 30129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:26:39.591030 30135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:39.591003 30136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:39.591938 30140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:26:39.592401 30129 server_base.cc:1034] running on GCE node
I20260810 23:26:39.593016 30129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:26:39.594046 30129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:26:39.595255 30129 hybrid_clock.cc:648] HybridClock initialized: now 1786404399595192 us; error 60 us; skew 500 ppm
I20260810 23:26:39.595571 30129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:26:39.597879 30129 webserver.cc:460] Webserver started at http://127.14.175.66:42723/ using document root <none> and password file <none>
I20260810 23:26:39.598563 30129 fs_manager.cc:359] Metadata directory not provided
I20260810 23:26:39.598667 30129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:26:39.598980 30129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:26:39.601889 30129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/data/instance:
uuid: "d02e53d988ab4e9f8131ed4f5c95fa5c"
format_stamp: "Formatted at 2026-08-10 23:26:39 on dist-test-slave-rmhg"
server_key: "1909da2115db2636bfca2a6267690efd"
server_key_iv: "dc2eb0d54e9a8f7fbc750cd707511198"
server_key_version: "encryptionkey@0"
I20260810 23:26:39.602675 30129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance:
uuid: "d02e53d988ab4e9f8131ed4f5c95fa5c"
format_stamp: "Formatted at 2026-08-10 23:26:39 on dist-test-slave-rmhg"
server_key: "1909da2115db2636bfca2a6267690efd"
server_key_iv: "dc2eb0d54e9a8f7fbc750cd707511198"
server_key_version: "encryptionkey@0"
I20260810 23:26:39.607867 30129 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 23:26:39.611244 30146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:39.612548 30129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:26:39.612715 30129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/wal
uuid: "d02e53d988ab4e9f8131ed4f5c95fa5c"
format_stamp: "Formatted at 2026-08-10 23:26:39 on dist-test-slave-rmhg"
server_key: "1909da2115db2636bfca2a6267690efd"
server_key_iv: "dc2eb0d54e9a8f7fbc750cd707511198"
server_key_version: "encryptionkey@0"
I20260810 23:26:39.612921 30129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:26:39.642055 30129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:26:39.643064 30129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:26:39.643389 30129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:26:39.644428 30129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:26:39.646162 30129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:26:39.646265 30129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:39.646374 30129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:26:39.646449 30129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:39.677922 30129 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.66:40309
I20260810 23:26:39.677992 30339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.66:40309 every 8 connection(s)
I20260810 23:26:39.679234 30129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
I20260810 23:26:39.686952 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 30129
I20260810 23:26:39.687142 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance
I20260810 23:26:39.687553 15037 external_mini_cluster.cc:1428] Setting key 3323f00b3ff10c1c95e000484d4324d7
I20260810 23:26:39.690486 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.67:0
--local_ip_for_outbound_sockets=127.14.175.67
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:36271
--builtin_ntp_servers=127.14.175.84:43507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 23:26:39.692569 30126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:39.681754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:46699 (local address 127.14.175.126:36271)
0810 23:26:39.681999 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:39.682006 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:39.683425 (+ 1419us) server_negotiation.cc:407] Connection header received
0810 23:26:39.685598 (+ 2173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:39.685603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:39.685677 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:39.685767 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:39.686884 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:39.687679 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:39.689842 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:39.690253 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:39.690369 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:39.691760 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:39.691765 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:39.691768 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:26:39.691787 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:39.691819 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:26:39.691831 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:39.691909 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:39.692136 (+ 227us) server_negotiation.cc:299] Negotiation successful
0810 23:26:39.692250 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260810 23:26:39.693727 30344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:39.682289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:36271 (local address 127.14.175.66:46699)
0810 23:26:39.683252 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:39.683302 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:39.685310 (+ 2008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:39.685963 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:39.685971 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:39.686066 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:39.686706 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:39.686724 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:39.687836 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:39.687841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:39.689696 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:39.689712 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:39.690478 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:39.690486 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:39.690757 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:39.691426 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:26:39.691429 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:39.691477 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:26:39.691562 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:39.691922 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:39.691934 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:39.691943 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 23:26:39.692117 (+ 174us) client_negotiation.cc:240] Negotiation successful
0810 23:26:39.692544 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":271,"threads_started":1}
I20260810 23:26:39.695181 30340 heartbeater.cc:347] Connected to a master server at 127.14.175.126:36271
I20260810 23:26:39.695472 30340 heartbeater.cc:464] Registering TS with master...
I20260810 23:26:39.696249 30340 heartbeater.cc:511] Master 127.14.175.126:36271 requested a full tablet report, sending...
I20260810 23:26:39.697729 29824 ts_manager.cc:194] Registered new tserver with Master: d02e53d988ab4e9f8131ed4f5c95fa5c (127.14.175.66:40309)
I20260810 23:26:39.698748 29824 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.175.66:46699
W20260810 23:26:40.039628 30345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:26:40.039965 30345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:26:40.040061 30345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:26:40.047442 30345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:26:40.047587 30345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:26:40.047756 30345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.67
I20260810 23:26:40.057085 30345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:43507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:36271
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.67
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:26:40.058451 30345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:26:40.060293 30345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:26:40.073427 30353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:40.073541 30357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:40.073465 30354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:26:40.074395 30345 server_base.cc:1034] running on GCE node
I20260810 23:26:40.075160 30345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:26:40.076263 30345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:26:40.077479 30345 hybrid_clock.cc:648] HybridClock initialized: now 1786404400077419 us; error 53 us; skew 500 ppm
I20260810 23:26:40.077875 30345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:26:40.080188 30345 webserver.cc:460] Webserver started at http://127.14.175.67:41731/ using document root <none> and password file <none>
I20260810 23:26:40.080914 30345 fs_manager.cc:359] Metadata directory not provided
I20260810 23:26:40.081022 30345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:26:40.081396 30345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:26:40.084420 30345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/data/instance:
uuid: "04349dd542d749d7bfdb3eae0f13dada"
format_stamp: "Formatted at 2026-08-10 23:26:40 on dist-test-slave-rmhg"
server_key: "788123caf56a2dc634c7ef2601f148f3"
server_key_iv: "77525ea852e0ac34a946f7258e03328d"
server_key_version: "encryptionkey@0"
I20260810 23:26:40.085328 30345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance:
uuid: "04349dd542d749d7bfdb3eae0f13dada"
format_stamp: "Formatted at 2026-08-10 23:26:40 on dist-test-slave-rmhg"
server_key: "788123caf56a2dc634c7ef2601f148f3"
server_key_iv: "77525ea852e0ac34a946f7258e03328d"
server_key_version: "encryptionkey@0"
I20260810 23:26:40.090741 30345 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260810 23:26:40.094452 30367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:40.095939 30345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:26:40.096104 30345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/wal
uuid: "04349dd542d749d7bfdb3eae0f13dada"
format_stamp: "Formatted at 2026-08-10 23:26:40 on dist-test-slave-rmhg"
server_key: "788123caf56a2dc634c7ef2601f148f3"
server_key_iv: "77525ea852e0ac34a946f7258e03328d"
server_key_version: "encryptionkey@0"
I20260810 23:26:40.096319 30345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:26:40.114750 30345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:26:40.115664 30345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:26:40.115965 30345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:26:40.116982 30345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:26:40.118592 30345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:26:40.118691 30345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:40.118819 30345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:26:40.118896 30345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:40.151674 30345 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.67:40879
I20260810 23:26:40.151732 30554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.67:40879 every 8 connection(s)
I20260810 23:26:40.153069 30345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
I20260810 23:26:40.159540 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 30345
I20260810 23:26:40.159796 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance
I20260810 23:26:40.160274 15037 external_mini_cluster.cc:1428] Setting key 52ab09e0df4007ec1eedc50c2bdb62d9
I20260810 23:26:40.166400 30126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:40.155812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:59653 (local address 127.14.175.126:36271)
0810 23:26:40.156035 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:40.156041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:40.157569 (+ 1528us) server_negotiation.cc:407] Connection header received
0810 23:26:40.159421 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:40.159428 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:40.159529 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:40.159665 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:40.160991 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:40.161939 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:40.163607 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:40.164010 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:40.164128 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:40.165567 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:40.165572 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:40.165575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:26:40.165596 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:40.165627 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:26:40.165640 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:40.165720 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:40.165942 (+ 222us) server_negotiation.cc:299] Negotiation successful
0810 23:26:40.166051 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 23:26:40.167512 30560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:40.156372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:36271 (local address 127.14.175.67:59653)
0810 23:26:40.157388 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:40.157443 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:40.159083 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:40.159978 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:40.159987 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:40.160112 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:40.160793 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:40.160815 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:40.162112 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:40.162117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:40.163439 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:40.163461 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:40.164252 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:40.164261 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:40.164513 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:40.165224 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:26:40.165227 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:40.165272 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:26:40.165361 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:40.165732 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:40.165743 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:40.165752 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 23:26:40.165918 (+ 166us) client_negotiation.cc:240] Negotiation successful
0810 23:26:40.166382 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":287,"threads_started":1}
I20260810 23:26:40.168918 30556 heartbeater.cc:347] Connected to a master server at 127.14.175.126:36271
I20260810 23:26:40.169269 30556 heartbeater.cc:464] Registering TS with master...
I20260810 23:26:40.170142 30556 heartbeater.cc:511] Master 127.14.175.126:36271 requested a full tablet report, sending...
I20260810 23:26:40.171696 29824 ts_manager.cc:194] Registered new tserver with Master: 04349dd542d749d7bfdb3eae0f13dada (127.14.175.67:40879)
I20260810 23:26:40.172501 29824 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.175.67:59653
I20260810 23:26:40.175889 15037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 23:26:40.182214 30126 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:40.180529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52822 (local address 127.14.175.126:36271)
0810 23:26:40.180765 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:40.180772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:40.181250 (+ 478us) server_negotiation.cc:407] Connection header received
0810 23:26:40.181396 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:40.181400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:40.181469 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:40.181555 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:40.182010 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52822: BlockingRecv error: recv got EOF from 127.0.0.1:52822 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 23:26:40.182922 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 29907
I20260810 23:26:40.191322 30576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:40.183909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59172 (local address 127.14.175.65:45763)
0810 23:26:40.184480 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:40.184487 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:40.184517 (+ 30us) server_negotiation.cc:407] Connection header received
0810 23:26:40.184581 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:40.184588 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:40.184818 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:40.184974 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:40.185449 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:40.186640 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:40.187944 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:40.188530 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:40.188652 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:40.189748 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:40.189759 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:40.189775 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:26:40.190441 (+ 666us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:40.190494 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:26:40.190517 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:40.190625 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:40.190797 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 23:26:40.190968 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":239,"threads_started":1}
I20260810 23:26:40.192790 30117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:26:40.273594 30122 heartbeater.cc:503] Master 127.14.175.126:36271 was elected leader, sending a full tablet report...
I20260810 23:26:40.275058 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 29907
W20260810 23:26:40.293071 29802 connection.cc:535] server connection from 127.14.175.65:53913 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 23:26:40.293377 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 30129
I20260810 23:26:40.301522 30578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:40.294304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52364 (local address 127.14.175.66:40309)
0810 23:26:40.294867 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:40.294874 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:40.294903 (+ 29us) server_negotiation.cc:407] Connection header received
0810 23:26:40.294967 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:40.294973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:40.295200 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:40.295336 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:40.295761 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:40.296773 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:40.298207 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:40.298794 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:40.298898 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:40.299981 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:40.299989 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:40.300004 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:26:40.300626 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:40.300703 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:26:40.300725 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:40.300816 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:40.301006 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 23:26:40.301129 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":253,"threads_started":1}
I20260810 23:26:40.302836 30332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:26:40.388353 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 30129
I20260810 23:26:40.412647 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 30345
I20260810 23:26:40.421015 30580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:40.413714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45500 (local address 127.14.175.67:40879)
0810 23:26:40.414332 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:40.414339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:40.414370 (+ 31us) server_negotiation.cc:407] Connection header received
0810 23:26:40.414432 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:40.414438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:40.414666 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:40.414812 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:40.415294 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:40.416359 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:40.417772 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:40.418370 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:40.418500 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:40.419561 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:40.419570 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:40.419585 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:26:40.420175 (+ 590us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:40.420224 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:26:40.420243 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:40.420348 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:40.420513 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 23:26:40.420668 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":304,"threads_started":1}
I20260810 23:26:40.422380 30544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:26:40.505105 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 30345
I20260810 23:26:40.531955 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 29775
I20260810 23:26:40.533318 29862 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:26:40.637811 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 29775
2026-08-10T23:26:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2602 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T23:26:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:26:40Z Disabled control of system clock
I20260810 23:26:40.728819 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:38621
--webserver_interface=127.14.175.126
--webserver_port=0
--builtin_ntp_servers=127.14.175.84:39635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:38621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 23:26:41.084395 30587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:26:41.084765 30587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:26:41.084892 30587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:26:41.092399 30587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:26:41.092531 30587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:26:41.092571 30587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:26:41.092643 30587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:26:41.092689 30587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:26:41.102109 30587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:39635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:38621
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:38621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:26:41.103441 30587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:26:41.105325 30587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:26:41.117115 30597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:41.117375 30596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:41.118616 30600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:26:41.119490 30587 server_base.cc:1034] running on GCE node
I20260810 23:26:41.120464 30587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:26:41.122227 30587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:26:41.123494 30587 hybrid_clock.cc:648] HybridClock initialized: now 1786404401123429 us; error 58 us; skew 500 ppm
I20260810 23:26:41.124027 30587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:26:41.126889 30587 webserver.cc:460] Webserver started at http://127.14.175.126:40387/ using document root <none> and password file <none>
I20260810 23:26:41.127727 30587 fs_manager.cc:359] Metadata directory not provided
I20260810 23:26:41.127830 30587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:26:41.128162 30587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:26:41.131667 30587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "924eb491ca814617af1cab776b80e04c"
format_stamp: "Formatted at 2026-08-10 23:26:41 on dist-test-slave-rmhg"
server_key: "2c0f29649df77491d29a51fbc250407c"
server_key_iv: "029a589dc58b676d76f0711e844bf637"
server_key_version: "encryptionkey@0"
I20260810 23:26:41.132570 30587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "924eb491ca814617af1cab776b80e04c"
format_stamp: "Formatted at 2026-08-10 23:26:41 on dist-test-slave-rmhg"
server_key: "2c0f29649df77491d29a51fbc250407c"
server_key_iv: "029a589dc58b676d76f0711e844bf637"
server_key_version: "encryptionkey@0"
I20260810 23:26:41.138443 30587 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 23:26:41.142412 30609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:41.143883 30587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:26:41.144069 30587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "924eb491ca814617af1cab776b80e04c"
format_stamp: "Formatted at 2026-08-10 23:26:41 on dist-test-slave-rmhg"
server_key: "2c0f29649df77491d29a51fbc250407c"
server_key_iv: "029a589dc58b676d76f0711e844bf637"
server_key_version: "encryptionkey@0"
I20260810 23:26:41.144299 30587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:26:41.150808 30587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:26:41.151708 30587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:26:41.152019 30587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:26:41.171022 30587 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:38621
I20260810 23:26:41.171041 30690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:38621 every 8 connection(s)
I20260810 23:26:41.172514 30587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:26:41.176847 30692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:26:41.178380 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 30587
I20260810 23:26:41.178574 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:26:41.179214 15037 external_mini_cluster.cc:1428] Setting key 0625034eb7dd5ebbf8b07bd1e87a6a56
I20260810 23:26:41.190105 30699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:41.181437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34354 (local address 127.14.175.126:38621)
0810 23:26:41.182753 (+ 1316us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:41.182771 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:41.182835 (+ 64us) server_negotiation.cc:407] Connection header received
0810 23:26:41.185367 (+ 2532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:41.185401 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:41.186054 (+ 653us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:41.186585 (+ 531us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:41.187761 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:41.187771 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:41.187791 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:26:41.187814 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:41.187902 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:26:41.187930 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:41.188190 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:41.188345 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 23:26:41.188706 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":423,"threads_started":1}
I20260810 23:26:41.191633 30692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 924eb491ca814617af1cab776b80e04c: Bootstrap starting.
I20260810 23:26:41.195111 30692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 924eb491ca814617af1cab776b80e04c: Neither blocks nor log segments found. Creating new log.
I20260810 23:26:41.196228 30692 log.cc:824] T 00000000000000000000000000000000 P 924eb491ca814617af1cab776b80e04c: Log is configured to *not* fsync() on all Append() calls
I20260810 23:26:41.199532 30692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 924eb491ca814617af1cab776b80e04c: No bootstrap required, opened a new log
I20260810 23:26:41.207887 30692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 924eb491ca814617af1cab776b80e04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924eb491ca814617af1cab776b80e04c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38621 } }
I20260810 23:26:41.208317 30692 raft_consensus.cc:374] T 00000000000000000000000000000000 P 924eb491ca814617af1cab776b80e04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:26:41.208429 30692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 924eb491ca814617af1cab776b80e04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 924eb491ca814617af1cab776b80e04c, State: Initialized, Role: FOLLOWER
I20260810 23:26:41.209040 30692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 924eb491ca814617af1cab776b80e04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924eb491ca814617af1cab776b80e04c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38621 } }
I20260810 23:26:41.209353 30692 raft_consensus.cc:388] T 00000000000000000000000000000000 P 924eb491ca814617af1cab776b80e04c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:26:41.209499 30692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 924eb491ca814617af1cab776b80e04c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:26:41.209667 30692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 924eb491ca814617af1cab776b80e04c [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:26:41.212409 30692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 924eb491ca814617af1cab776b80e04c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924eb491ca814617af1cab776b80e04c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38621 } }
I20260810 23:26:41.213042 30692 leader_election.cc:302] T 00000000000000000000000000000000 P 924eb491ca814617af1cab776b80e04c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 924eb491ca814617af1cab776b80e04c; no voters:
I20260810 23:26:41.213723 30692 leader_election.cc:288] T 00000000000000000000000000000000 P 924eb491ca814617af1cab776b80e04c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:26:41.213869 30703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 924eb491ca814617af1cab776b80e04c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:26:41.214608 30703 raft_consensus.cc:686] T 00000000000000000000000000000000 P 924eb491ca814617af1cab776b80e04c [term 1 LEADER]: Becoming Leader. State: Replica: 924eb491ca814617af1cab776b80e04c, State: Running, Role: LEADER
I20260810 23:26:41.215232 30703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 924eb491ca814617af1cab776b80e04c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924eb491ca814617af1cab776b80e04c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38621 } }
I20260810 23:26:41.215874 30692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 924eb491ca814617af1cab776b80e04c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:26:41.218554 30705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 924eb491ca814617af1cab776b80e04c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "924eb491ca814617af1cab776b80e04c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924eb491ca814617af1cab776b80e04c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38621 } } }
I20260810 23:26:41.219012 30705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 924eb491ca814617af1cab776b80e04c [sys.catalog]: This master's current role is: LEADER
I20260810 23:26:41.219386 30708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 924eb491ca814617af1cab776b80e04c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 924eb491ca814617af1cab776b80e04c. Latest consensus state: current_term: 1 leader_uuid: "924eb491ca814617af1cab776b80e04c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924eb491ca814617af1cab776b80e04c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38621 } } }
I20260810 23:26:41.219735 30708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 924eb491ca814617af1cab776b80e04c [sys.catalog]: This master's current role is: LEADER
I20260810 23:26:41.222215 30718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:26:41.226927 30718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:26:41.236797 30718 catalog_manager.cc:1347] Generated new cluster ID: 3306b308025a4b71bfdbb4b97067b578
I20260810 23:26:41.236995 30718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:26:41.251760 30718 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:26:41.252303 30718 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:26:41.267127 30718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 924eb491ca814617af1cab776b80e04c: Generated new TSK 0
I20260810 23:26:41.268404 30718 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:26:41.275446 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.65:0
--local_ip_for_outbound_sockets=127.14.175.65
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:38621
--builtin_ntp_servers=127.14.175.84:39635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 23:26:41.626322 30733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:26:41.626643 30733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:26:41.626742 30733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:26:41.634171 30733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:26:41.634294 30733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:26:41.634404 30733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.65
I20260810 23:26:41.643827 30733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:39635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:38621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.65
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:26:41.645213 30733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:26:41.647119 30733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:26:41.661237 30741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:41.661135 30746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:41.661191 30742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:26:41.662079 30733 server_base.cc:1034] running on GCE node
I20260810 23:26:41.662755 30733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:26:41.663779 30733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:26:41.665032 30733 hybrid_clock.cc:648] HybridClock initialized: now 1786404401664970 us; error 51 us; skew 500 ppm
I20260810 23:26:41.665418 30733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:26:41.673261 30733 webserver.cc:460] Webserver started at http://127.14.175.65:33923/ using document root <none> and password file <none>
I20260810 23:26:41.674108 30733 fs_manager.cc:359] Metadata directory not provided
I20260810 23:26:41.674243 30733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:26:41.674626 30733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:26:41.677788 30733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/data/instance:
uuid: "db180eec26fb43a2a5b94d5b79dc6828"
format_stamp: "Formatted at 2026-08-10 23:26:41 on dist-test-slave-rmhg"
server_key: "e0b140f2d3ff7baa7e6695a2a9751957"
server_key_iv: "c8000e05c06760262254fae26d6af07a"
server_key_version: "encryptionkey@0"
I20260810 23:26:41.678638 30733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance:
uuid: "db180eec26fb43a2a5b94d5b79dc6828"
format_stamp: "Formatted at 2026-08-10 23:26:41 on dist-test-slave-rmhg"
server_key: "e0b140f2d3ff7baa7e6695a2a9751957"
server_key_iv: "c8000e05c06760262254fae26d6af07a"
server_key_version: "encryptionkey@0"
I20260810 23:26:41.684266 30733 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:26:41.688239 30761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:41.689939 30733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:26:41.690129 30733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/wal
uuid: "db180eec26fb43a2a5b94d5b79dc6828"
format_stamp: "Formatted at 2026-08-10 23:26:41 on dist-test-slave-rmhg"
server_key: "e0b140f2d3ff7baa7e6695a2a9751957"
server_key_iv: "c8000e05c06760262254fae26d6af07a"
server_key_version: "encryptionkey@0"
I20260810 23:26:41.690374 30733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:26:41.696655 30733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:26:41.697647 30733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:26:41.697928 30733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:26:41.698915 30733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:26:41.700454 30733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:26:41.700559 30733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:41.700682 30733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:26:41.700754 30733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:41.734229 30733 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.65:42281
I20260810 23:26:41.734266 30949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.65:42281 every 8 connection(s)
I20260810 23:26:41.735689 30733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
I20260810 23:26:41.744689 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:41.738777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:50143 (local address 127.14.175.126:38621)
0810 23:26:41.739296 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:41.739302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:41.740618 (+ 1316us) server_negotiation.cc:407] Connection header received
0810 23:26:41.742408 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:41.742412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:41.742475 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:41.742579 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:41.743895 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:41.743900 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:41.743903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:26:41.743921 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:41.743968 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:26:41.743980 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:41.744085 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:41.744261 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 23:26:41.744334 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":244,"threads_started":1}
I20260810 23:26:41.744764 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 30733
I20260810 23:26:41.744943 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance
I20260810 23:26:41.745630 30955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:41.739397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:38621 (local address 127.14.175.65:50143)
0810 23:26:41.740429 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:41.740480 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:41.742141 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:41.742787 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:41.742796 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:41.742899 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:41.743581 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:26:41.743584 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:41.743627 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:26:41.743723 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:41.744072 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:41.744081 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:41.744087 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 23:26:41.744312 (+ 225us) client_negotiation.cc:240] Negotiation successful
0810 23:26:41.744562 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":293,"threads_started":1}
I20260810 23:26:41.745645 15037 external_mini_cluster.cc:1428] Setting key ca9b6ad8f9d55180544cbf88835f337d
I20260810 23:26:41.747278 30950 heartbeater.cc:347] Connected to a master server at 127.14.175.126:38621
I20260810 23:26:41.747589 30950 heartbeater.cc:464] Registering TS with master...
I20260810 23:26:41.748250 30950 heartbeater.cc:511] Master 127.14.175.126:38621 requested a full tablet report, sending...
I20260810 23:26:41.748875 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.66:0
--local_ip_for_outbound_sockets=127.14.175.66
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:38621
--builtin_ntp_servers=127.14.175.84:39635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 23:26:41.750440 30633 ts_manager.cc:194] Registered new tserver with Master: db180eec26fb43a2a5b94d5b79dc6828 (127.14.175.65:42281)
W20260810 23:26:42.096495 30958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:26:42.096793 30958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:26:42.096911 30958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:26:42.104380 30958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:26:42.104524 30958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:26:42.104671 30958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.66
I20260810 23:26:42.114173 30958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:39635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:38621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.66
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:26:42.115538 30958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:26:42.117568 30958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:26:42.131647 30963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:42.131616 30964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:42.131690 30967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:26:42.132598 30958 server_base.cc:1034] running on GCE node
I20260810 23:26:42.133323 30958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:26:42.134393 30958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:26:42.135622 30958 hybrid_clock.cc:648] HybridClock initialized: now 1786404402135564 us; error 51 us; skew 500 ppm
I20260810 23:26:42.135972 30958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:26:42.138695 30958 webserver.cc:460] Webserver started at http://127.14.175.66:40023/ using document root <none> and password file <none>
I20260810 23:26:42.139482 30958 fs_manager.cc:359] Metadata directory not provided
I20260810 23:26:42.139606 30958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:26:42.139940 30958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:26:42.143158 30958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/data/instance:
uuid: "1a723480798d465f97e5a1c094a0fcb2"
format_stamp: "Formatted at 2026-08-10 23:26:42 on dist-test-slave-rmhg"
server_key: "1c16b6c60166d2a3c916c2eeed490972"
server_key_iv: "1c19341d611aef79185f2ffc9213d563"
server_key_version: "encryptionkey@0"
I20260810 23:26:42.144096 30958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance:
uuid: "1a723480798d465f97e5a1c094a0fcb2"
format_stamp: "Formatted at 2026-08-10 23:26:42 on dist-test-slave-rmhg"
server_key: "1c16b6c60166d2a3c916c2eeed490972"
server_key_iv: "1c19341d611aef79185f2ffc9213d563"
server_key_version: "encryptionkey@0"
I20260810 23:26:42.150328 30958 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260810 23:26:42.154346 30978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:42.155920 30958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:26:42.156121 30958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/wal
uuid: "1a723480798d465f97e5a1c094a0fcb2"
format_stamp: "Formatted at 2026-08-10 23:26:42 on dist-test-slave-rmhg"
server_key: "1c16b6c60166d2a3c916c2eeed490972"
server_key_iv: "1c19341d611aef79185f2ffc9213d563"
server_key_version: "encryptionkey@0"
I20260810 23:26:42.156378 30958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:26:42.162437 30958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:26:42.163362 30958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:26:42.163642 30958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:26:42.164700 30958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:26:42.166519 30958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:26:42.166631 30958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:42.166749 30958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:26:42.166818 30958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:42.198385 30958 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.66:42857
I20260810 23:26:42.198438 31166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.66:42857 every 8 connection(s)
I20260810 23:26:42.199764 30958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
I20260810 23:26:42.207995 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:42.202494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:39597 (local address 127.14.175.126:38621)
0810 23:26:42.202707 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:42.202712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:42.204204 (+ 1492us) server_negotiation.cc:407] Connection header received
0810 23:26:42.205984 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:42.205989 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:42.206065 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:42.206156 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:42.207393 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:42.207398 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:42.207401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:26:42.207418 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:42.207452 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:26:42.207464 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:42.207560 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:42.207723 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 23:26:42.207776 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 23:26:42.208333 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 30958
I20260810 23:26:42.208519 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance
I20260810 23:26:42.209090 15037 external_mini_cluster.cc:1428] Setting key 363c9cec2b4cf889e33ce8c4c7632358
I20260810 23:26:42.209349 31171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:42.203022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:38621 (local address 127.14.175.66:39597)
0810 23:26:42.204016 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:42.204063 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:42.205701 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:42.206343 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:42.206350 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:42.206446 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:42.207093 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:26:42.207095 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:42.207136 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:26:42.207229 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:42.207535 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:42.207541 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:42.207546 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:26:42.207788 (+ 242us) client_negotiation.cc:240] Negotiation successful
0810 23:26:42.208057 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":288,"threads_started":1}
I20260810 23:26:42.210760 31167 heartbeater.cc:347] Connected to a master server at 127.14.175.126:38621
I20260810 23:26:42.211081 31167 heartbeater.cc:464] Registering TS with master...
I20260810 23:26:42.211819 31167 heartbeater.cc:511] Master 127.14.175.126:38621 requested a full tablet report, sending...
I20260810 23:26:42.212424 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.67:0
--local_ip_for_outbound_sockets=127.14.175.67
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:38621
--builtin_ntp_servers=127.14.175.84:39635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 23:26:42.213510 30634 ts_manager.cc:194] Registered new tserver with Master: 1a723480798d465f97e5a1c094a0fcb2 (127.14.175.66:42857)
W20260810 23:26:42.558358 31172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:26:42.558666 31172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:26:42.558799 31172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:26:42.566241 31172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:26:42.566363 31172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:26:42.566510 31172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.67
I20260810 23:26:42.575927 31172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:39635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:38621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.67
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:26:42.577441 31172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:26:42.579650 31172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:26:42.593801 31183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:42.593868 31188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:42.593894 31182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:26:42.594918 31172 server_base.cc:1034] running on GCE node
I20260810 23:26:42.595674 31172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:26:42.596813 31172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:26:42.598019 31172 hybrid_clock.cc:648] HybridClock initialized: now 1786404402597956 us; error 64 us; skew 500 ppm
I20260810 23:26:42.598371 31172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:26:42.600931 31172 webserver.cc:460] Webserver started at http://127.14.175.67:45673/ using document root <none> and password file <none>
I20260810 23:26:42.601733 31172 fs_manager.cc:359] Metadata directory not provided
I20260810 23:26:42.601837 31172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:26:42.602180 31172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:26:42.605327 31172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/data/instance:
uuid: "dc171503403d445d97d7eb34ecdb5e2c"
format_stamp: "Formatted at 2026-08-10 23:26:42 on dist-test-slave-rmhg"
server_key: "fd4d126fb1f8eed654e127a06dcd1cce"
server_key_iv: "b90eefe216dacd4fa53e3c85f4182251"
server_key_version: "encryptionkey@0"
I20260810 23:26:42.606186 31172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance:
uuid: "dc171503403d445d97d7eb34ecdb5e2c"
format_stamp: "Formatted at 2026-08-10 23:26:42 on dist-test-slave-rmhg"
server_key: "fd4d126fb1f8eed654e127a06dcd1cce"
server_key_iv: "b90eefe216dacd4fa53e3c85f4182251"
server_key_version: "encryptionkey@0"
I20260810 23:26:42.611730 31172 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260810 23:26:42.615463 31196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:42.617106 31172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:26:42.617344 31172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/wal
uuid: "dc171503403d445d97d7eb34ecdb5e2c"
format_stamp: "Formatted at 2026-08-10 23:26:42 on dist-test-slave-rmhg"
server_key: "fd4d126fb1f8eed654e127a06dcd1cce"
server_key_iv: "b90eefe216dacd4fa53e3c85f4182251"
server_key_version: "encryptionkey@0"
I20260810 23:26:42.617573 31172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:26:42.623682 31172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:26:42.624568 31172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:26:42.624837 31172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:26:42.625837 31172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:26:42.627342 31172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:26:42.627442 31172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:42.627555 31172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:26:42.627624 31172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:42.659101 31172 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.67:46387
I20260810 23:26:42.659152 31376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.67:46387 every 8 connection(s)
I20260810 23:26:42.660534 31172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
I20260810 23:26:42.661273 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 31172
I20260810 23:26:42.661464 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance
I20260810 23:26:42.662066 15037 external_mini_cluster.cc:1428] Setting key d76738459bd2c4fc7ecb0d8a47e736e4
I20260810 23:26:42.669449 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:42.663731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:43343 (local address 127.14.175.126:38621)
0810 23:26:42.663935 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:42.663941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:42.665799 (+ 1858us) server_negotiation.cc:407] Connection header received
0810 23:26:42.667442 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:42.667446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:42.667509 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:42.667594 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:42.668831 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:42.668836 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:42.668839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:26:42.668854 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:42.668887 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:26:42.668899 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:42.668991 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:42.669139 (+ 148us) server_negotiation.cc:299] Negotiation successful
0810 23:26:42.669225 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:26:42.670392 31380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:42.664400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:38621 (local address 127.14.175.67:43343)
0810 23:26:42.665627 (+ 1227us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:42.665679 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:42.667207 (+ 1528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:42.667790 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:42.667798 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:42.667892 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:42.668545 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:26:42.668547 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:42.668590 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:26:42.668679 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:42.668972 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:42.668978 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:42.668983 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:26:42.669248 (+ 265us) client_negotiation.cc:240] Negotiation successful
0810 23:26:42.669461 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":331,"threads_started":1}
I20260810 23:26:42.671705 31378 heartbeater.cc:347] Connected to a master server at 127.14.175.126:38621
I20260810 23:26:42.671993 31378 heartbeater.cc:464] Registering TS with master...
I20260810 23:26:42.672580 31378 heartbeater.cc:511] Master 127.14.175.126:38621 requested a full tablet report, sending...
I20260810 23:26:42.673981 30634 ts_manager.cc:194] Registered new tserver with Master: dc171503403d445d97d7eb34ecdb5e2c (127.14.175.67:46387)
I20260810 23:26:42.677733 15037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 23:26:42.683702 30952 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:26:42.682166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34366 (local address 127.14.175.126:38621)
0810 23:26:42.682394 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:42.682401 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:42.682821 (+ 420us) server_negotiation.cc:407] Connection header received
0810 23:26:42.682967 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:42.682972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:42.683025 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:42.683099 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:42.683414 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34366: BlockingRecv error: recv got EOF from 127.0.0.1:34366 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:26:42.685240 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 30733
I20260810 23:26:42.688496 31396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:42.686112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58008 (local address 127.14.175.65:42281)
0810 23:26:42.686600 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:42.686607 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:42.686635 (+ 28us) server_negotiation.cc:407] Connection header received
0810 23:26:42.686706 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:42.686712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:42.686927 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:42.687053 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:42.687900 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:42.687906 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:42.687917 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:26:42.687929 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:42.687970 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:26:42.687987 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:42.688090 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:42.688163 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 23:26:42.688254 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":209,"threads_started":1}
I20260810 23:26:42.689774 30945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:26:42.770888 30950 heartbeater.cc:503] Master 127.14.175.126:38621 was elected leader, sending a full tablet report...
I20260810 23:26:42.772436 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 30733
W20260810 23:26:42.799230 30612 connection.cc:535] server connection from 127.14.175.65:50143 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 23:26:42.799451 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 30958
I20260810 23:26:42.803004 31398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:42.800469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36126 (local address 127.14.175.66:42857)
0810 23:26:42.801058 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:42.801068 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:42.801103 (+ 35us) server_negotiation.cc:407] Connection header received
0810 23:26:42.801198 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:42.801207 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:42.801438 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:42.801602 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:42.802360 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:42.802366 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:42.802378 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:26:42.802399 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:42.802437 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:26:42.802451 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:42.802556 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:42.802622 (+ 66us) server_negotiation.cc:299] Negotiation successful
0810 23:26:42.802710 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":305,"threads_started":1}
I20260810 23:26:42.804222 31158 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:26:42.885924 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 30958
I20260810 23:26:42.908401 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 31172
I20260810 23:26:42.912495 31403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:42.909527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48240 (local address 127.14.175.67:46387)
0810 23:26:42.910244 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:42.910251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:42.910284 (+ 33us) server_negotiation.cc:407] Connection header received
0810 23:26:42.910355 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:42.910361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:42.910579 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:42.910708 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:42.911742 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:42.911749 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:42.911761 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:26:42.911778 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:42.911817 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:26:42.911830 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:42.911965 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:42.912065 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 23:26:42.912168 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":374,"threads_started":1}
I20260810 23:26:42.913931 31370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:26:42.996327 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 31172
I20260810 23:26:43.022716 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 30587
I20260810 23:26:43.023988 30684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:26:43.128217 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 30587
2026-08-10T23:26:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2490 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:26:43 dist-test-slave-rmhg krb5kdc[31410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:26:43 dist-test-slave-rmhg krb5kdc[31410](info): set up 2 sockets
Aug 10 23:26:43 dist-test-slave-rmhg krb5kdc[31410](info): commencing operation
krb5kdc: starting...
W20260810 23:26:44.404101 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.228s user 0.000s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:26:44 dist-test-slave-rmhg krb5kdc[31410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:26:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:26:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
I20260810 23:26:44.696120 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:33295
--webserver_interface=127.14.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:38061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:33295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:26:45.053005 31437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:26:45.053339 31437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:26:45.053442 31437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:26:45.060838 31437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:26:45.060963 31437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:26:45.061010 31437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:26:45.061069 31437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:26:45.061112 31437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:26:45.071488 31437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:38061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:33295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:33295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:26:45.072888 31437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:26:45.074296 31437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:26:45.086032 31442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:45.086126 31444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:45.086087 31448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:26:45.087467 31437 server_base.cc:1034] running on GCE node
I20260810 23:26:45.088361 31437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:26:45.090015 31437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:26:45.091272 31437 hybrid_clock.cc:648] HybridClock initialized: now 1786404405091210 us; error 49 us; skew 500 ppm
I20260810 23:26:45.091694 31437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:26:45 dist-test-slave-rmhg krb5kdc[31410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404405, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:26:45.095022 31437 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:26:45.096721 31437 webserver.cc:460] Webserver started at http://127.14.175.126:36249/ using document root <none> and password file <none>
I20260810 23:26:45.097491 31437 fs_manager.cc:359] Metadata directory not provided
I20260810 23:26:45.097630 31437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:26:45.097944 31437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:26:45.101138 31437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "4bda6b8aac404bd3898b9fe4fbd12a82"
format_stamp: "Formatted at 2026-08-10 23:26:45 on dist-test-slave-rmhg"
server_key: "4cb873780633c4ddaa06daca0c9d778e"
server_key_iv: "db6b64d9fd53864ad229472ea7c22379"
server_key_version: "encryptionkey@0"
I20260810 23:26:45.101974 31437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "4bda6b8aac404bd3898b9fe4fbd12a82"
format_stamp: "Formatted at 2026-08-10 23:26:45 on dist-test-slave-rmhg"
server_key: "4cb873780633c4ddaa06daca0c9d778e"
server_key_iv: "db6b64d9fd53864ad229472ea7c22379"
server_key_version: "encryptionkey@0"
I20260810 23:26:45.107575 31437 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:26:45.111428 31459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:45.112888 31437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 23:26:45.113076 31437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "4bda6b8aac404bd3898b9fe4fbd12a82"
format_stamp: "Formatted at 2026-08-10 23:26:45 on dist-test-slave-rmhg"
server_key: "4cb873780633c4ddaa06daca0c9d778e"
server_key_iv: "db6b64d9fd53864ad229472ea7c22379"
server_key_version: "encryptionkey@0"
I20260810 23:26:45.113353 31437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:26:45.131102 31437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:26:45.135802 31437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:26:45.136114 31437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:26:45.155232 31437 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:33295
I20260810 23:26:45.155273 31543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:33295 every 8 connection(s)
I20260810 23:26:45.156687 31437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:26:45.160573 31544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:26:45.165180 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 31437
I20260810 23:26:45.165414 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:26:45.166106 15037 external_mini_cluster.cc:1428] Setting key 669259522c19eef7802cf0e026b75da4
I20260810 23:26:45.172653 31544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bda6b8aac404bd3898b9fe4fbd12a82: Bootstrap starting.
I20260810 23:26:45.176589 31544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bda6b8aac404bd3898b9fe4fbd12a82: Neither blocks nor log segments found. Creating new log.
I20260810 23:26:45.177874 31544 log.cc:824] T 00000000000000000000000000000000 P 4bda6b8aac404bd3898b9fe4fbd12a82: Log is configured to *not* fsync() on all Append() calls
I20260810 23:26:45.181794 31544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bda6b8aac404bd3898b9fe4fbd12a82: No bootstrap required, opened a new log
Aug 10 23:26:45 dist-test-slave-rmhg krb5kdc[31410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:26:45.192646 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:45.168200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52142 (local address 127.14.175.126:33295)
0810 23:26:45.169005 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:45.169019 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:45.169066 (+ 47us) server_negotiation.cc:407] Connection header received
0810 23:26:45.171178 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:45.171207 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:45.171650 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:45.172002 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:45.173919 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:45.175307 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:45.177097 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:45.177886 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:45.178066 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:45.183049 (+ 4983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:45.183063 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:45.183087 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:45.183140 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:45.187457 (+ 4317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:45.188334 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:45.188346 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:45.188360 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:45.188615 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:45.188972 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:45.188977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:45.188980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:45.189843 (+ 863us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:45.190200 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:45.190568 (+ 368us) server_negotiation.cc:299] Negotiation successful
0810 23:26:45.191109 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":355,"threads_started":1}
I20260810 23:26:45.192625 31544 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bda6b8aac404bd3898b9fe4fbd12a82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bda6b8aac404bd3898b9fe4fbd12a82" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 33295 } }
I20260810 23:26:45.193176 31544 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bda6b8aac404bd3898b9fe4fbd12a82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:26:45.193293 31544 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bda6b8aac404bd3898b9fe4fbd12a82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bda6b8aac404bd3898b9fe4fbd12a82, State: Initialized, Role: FOLLOWER
I20260810 23:26:45.193961 31544 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bda6b8aac404bd3898b9fe4fbd12a82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bda6b8aac404bd3898b9fe4fbd12a82" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 33295 } }
I20260810 23:26:45.194204 31544 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bda6b8aac404bd3898b9fe4fbd12a82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:26:45.194365 31544 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bda6b8aac404bd3898b9fe4fbd12a82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:26:45.194573 31544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bda6b8aac404bd3898b9fe4fbd12a82 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:26:45.197558 31544 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bda6b8aac404bd3898b9fe4fbd12a82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bda6b8aac404bd3898b9fe4fbd12a82" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 33295 } }
I20260810 23:26:45.198062 31544 leader_election.cc:302] T 00000000000000000000000000000000 P 4bda6b8aac404bd3898b9fe4fbd12a82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bda6b8aac404bd3898b9fe4fbd12a82; no voters:
I20260810 23:26:45.198674 31544 leader_election.cc:288] T 00000000000000000000000000000000 P 4bda6b8aac404bd3898b9fe4fbd12a82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:26:45.198804 31552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bda6b8aac404bd3898b9fe4fbd12a82 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:26:45.200070 31552 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bda6b8aac404bd3898b9fe4fbd12a82 [term 1 LEADER]: Becoming Leader. State: Replica: 4bda6b8aac404bd3898b9fe4fbd12a82, State: Running, Role: LEADER
I20260810 23:26:45.200708 31552 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bda6b8aac404bd3898b9fe4fbd12a82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bda6b8aac404bd3898b9fe4fbd12a82" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 33295 } }
I20260810 23:26:45.200941 31544 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bda6b8aac404bd3898b9fe4fbd12a82 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:26:45.203702 31554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bda6b8aac404bd3898b9fe4fbd12a82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bda6b8aac404bd3898b9fe4fbd12a82. Latest consensus state: current_term: 1 leader_uuid: "4bda6b8aac404bd3898b9fe4fbd12a82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bda6b8aac404bd3898b9fe4fbd12a82" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 33295 } } }
I20260810 23:26:45.204202 31554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bda6b8aac404bd3898b9fe4fbd12a82 [sys.catalog]: This master's current role is: LEADER
I20260810 23:26:45.204001 31553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bda6b8aac404bd3898b9fe4fbd12a82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bda6b8aac404bd3898b9fe4fbd12a82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bda6b8aac404bd3898b9fe4fbd12a82" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 33295 } } }
I20260810 23:26:45.204326 31553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bda6b8aac404bd3898b9fe4fbd12a82 [sys.catalog]: This master's current role is: LEADER
I20260810 23:26:45.205235 31566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:26:45.211406 31566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:26:45.221717 31566 catalog_manager.cc:1347] Generated new cluster ID: e4d8f431ed904d8296668da9765a9549
I20260810 23:26:45.221869 31566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:26:45.244395 31566 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:26:45.245993 31566 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:26:45.262065 31566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bda6b8aac404bd3898b9fe4fbd12a82: Generated new TSK 0
I20260810 23:26:45.263015 31566 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
I20260810 23:26:45.339541 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.65:0
--local_ip_for_outbound_sockets=127.14.175.65
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:33295
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:38061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:26:45.691761 31581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:26:45.692091 31581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:26:45.692236 31581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:26:45.699877 31581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:26:45.700037 31581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:26:45.700188 31581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.65
I20260810 23:26:45.710999 31581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:38061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:33295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.65
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:26:45.712443 31581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:26:45.713971 31581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:26:45.728088 31596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:45.728045 31594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:45.728157 31592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:26:45.730727 31581 server_base.cc:1034] running on GCE node
I20260810 23:26:45.731424 31581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:26:45.732463 31581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:26:45.733753 31581 hybrid_clock.cc:648] HybridClock initialized: now 1786404405733696 us; error 50 us; skew 500 ppm
I20260810 23:26:45.734126 31581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:26:45 dist-test-slave-rmhg krb5kdc[31410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404405, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:26:45.737751 31581 init.cc:377] Logged in from keytab as kudu/127.14.175.65@KRBTEST.COM (short username kudu)
I20260810 23:26:45.739576 31581 webserver.cc:460] Webserver started at http://127.14.175.65:36937/ using document root <none> and password file <none>
I20260810 23:26:45.740399 31581 fs_manager.cc:359] Metadata directory not provided
I20260810 23:26:45.740543 31581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:26:45.740859 31581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:26:45.744428 31581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/data/instance:
uuid: "ae494906d89e46548c3c8545331c59a3"
format_stamp: "Formatted at 2026-08-10 23:26:45 on dist-test-slave-rmhg"
server_key: "098242c00a34084915f45329c2375c9a"
server_key_iv: "e642ecb71f2df4effb4141c2738bccfa"
server_key_version: "encryptionkey@0"
I20260810 23:26:45.745450 31581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance:
uuid: "ae494906d89e46548c3c8545331c59a3"
format_stamp: "Formatted at 2026-08-10 23:26:45 on dist-test-slave-rmhg"
server_key: "098242c00a34084915f45329c2375c9a"
server_key_iv: "e642ecb71f2df4effb4141c2738bccfa"
server_key_version: "encryptionkey@0"
I20260810 23:26:45.751189 31581 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:26:45.755110 31609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:45.756699 31581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:26:45.756886 31581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/wal
uuid: "ae494906d89e46548c3c8545331c59a3"
format_stamp: "Formatted at 2026-08-10 23:26:45 on dist-test-slave-rmhg"
server_key: "098242c00a34084915f45329c2375c9a"
server_key_iv: "e642ecb71f2df4effb4141c2738bccfa"
server_key_version: "encryptionkey@0"
I20260810 23:26:45.757103 31581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:26:45.779297 31581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:26:45.784008 31581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:26:45.784332 31581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:26:45.785501 31581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:26:45.787107 31581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:26:45.787209 31581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:45.787343 31581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:26:45.787398 31581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:45.820645 31581 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.65:45495
I20260810 23:26:45.820694 31786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.65:45495 every 8 connection(s)
I20260810 23:26:45.822048 31581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
I20260810 23:26:45.830188 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 31581
I20260810 23:26:45.830417 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance
I20260810 23:26:45.831288 15037 external_mini_cluster.cc:1428] Setting key 23a868ea201e22633fde7903e81d76b0
Aug 10 23:26:45 dist-test-slave-rmhg krb5kdc[31410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404405, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:26:45.844784 31790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:45.824801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:43817 (local address 127.14.175.126:33295)
0810 23:26:45.825374 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:45.825381 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:45.826290 (+ 909us) server_negotiation.cc:407] Connection header received
0810 23:26:45.828195 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:45.828199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:45.828272 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:45.828372 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:45.830046 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:45.831283 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:45.833554 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:45.834216 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:45.834368 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:45.838007 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:45.838014 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:45.838019 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:45.838061 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:45.841183 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:45.842124 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:45.842130 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:45.842133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:45.842288 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:45.842802 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:45.842807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:45.842810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:45.843161 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:45.843314 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:45.844170 (+ 856us) server_negotiation.cc:299] Negotiation successful
0810 23:26:45.844407 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":264,"threads_started":1}
I20260810 23:26:45.845973 31795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:45.825378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:33295 (local address 127.14.175.65:43817)
0810 23:26:45.826119 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:45.826161 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:45.827882 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:45.828607 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:45.828615 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:45.829199 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:45.829836 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:45.829856 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:45.831488 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:45.831493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:45.833386 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:45.833404 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:45.834437 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:45.834443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:45.834689 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:45.835367 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:45.835392 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:45.837726 (+ 2334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:45.841369 (+ 3643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:45.841378 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:45.841395 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:45.841964 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:45.842440 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:45.842444 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:45.842447 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:45.842673 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:45.843333 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:45.843341 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:45.843741 (+ 400us) client_negotiation.cc:769] Sending connection context
0810 23:26:45.844025 (+ 284us) client_negotiation.cc:240] Negotiation successful
0810 23:26:45.844510 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":264,"threads_started":1}
I20260810 23:26:45.847750 31787 heartbeater.cc:347] Connected to a master server at 127.14.175.126:33295
I20260810 23:26:45.848057 31787 heartbeater.cc:464] Registering TS with master...
I20260810 23:26:45.848862 31787 heartbeater.cc:511] Master 127.14.175.126:33295 requested a full tablet report, sending...
I20260810 23:26:45.851366 31480 ts_manager.cc:194] Registered new tserver with Master: ae494906d89e46548c3c8545331c59a3 (127.14.175.65:45495)
I20260810 23:26:45.853447 31480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.65@KRBTEST.COM'} at 127.14.175.65:43817
WARNING: no policy specified for kudu/127.14.175.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
I20260810 23:26:45.895946 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.66:0
--local_ip_for_outbound_sockets=127.14.175.66
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:33295
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:38061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:26:46.244728 31799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:26:46.245034 31799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:26:46.245203 31799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:26:46.252588 31799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:26:46.252727 31799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:26:46.252887 31799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.66
I20260810 23:26:46.263247 31799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:38061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:33295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.66
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:26:46.264693 31799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:26:46.266194 31799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:26:46.279722 31808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:46.279790 31809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:46.279767 31811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:26:46.281009 31799 server_base.cc:1034] running on GCE node
I20260810 23:26:46.281723 31799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:26:46.282732 31799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:26:46.283972 31799 hybrid_clock.cc:648] HybridClock initialized: now 1786404406283887 us; error 77 us; skew 500 ppm
I20260810 23:26:46.284310 31799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:26:46 dist-test-slave-rmhg krb5kdc[31410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404406, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:26:46.287665 31799 init.cc:377] Logged in from keytab as kudu/127.14.175.66@KRBTEST.COM (short username kudu)
I20260810 23:26:46.289403 31799 webserver.cc:460] Webserver started at http://127.14.175.66:41321/ using document root <none> and password file <none>
I20260810 23:26:46.290154 31799 fs_manager.cc:359] Metadata directory not provided
I20260810 23:26:46.290288 31799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:26:46.290601 31799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:26:46.293808 31799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/data/instance:
uuid: "9e2d6b54bccf4e94a495e31ae2e0307c"
format_stamp: "Formatted at 2026-08-10 23:26:46 on dist-test-slave-rmhg"
server_key: "a35630803c7647966b1b9b2a1b64258f"
server_key_iv: "a43b1b2a3ea279d015178ac88fe0dcfd"
server_key_version: "encryptionkey@0"
I20260810 23:26:46.294921 31799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance:
uuid: "9e2d6b54bccf4e94a495e31ae2e0307c"
format_stamp: "Formatted at 2026-08-10 23:26:46 on dist-test-slave-rmhg"
server_key: "a35630803c7647966b1b9b2a1b64258f"
server_key_iv: "a43b1b2a3ea279d015178ac88fe0dcfd"
server_key_version: "encryptionkey@0"
I20260810 23:26:46.300524 31799 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 23:26:46.304598 31819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:46.306252 31799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 23:26:46.306442 31799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/wal
uuid: "9e2d6b54bccf4e94a495e31ae2e0307c"
format_stamp: "Formatted at 2026-08-10 23:26:46 on dist-test-slave-rmhg"
server_key: "a35630803c7647966b1b9b2a1b64258f"
server_key_iv: "a43b1b2a3ea279d015178ac88fe0dcfd"
server_key_version: "encryptionkey@0"
I20260810 23:26:46.306666 31799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:26:46.327896 31799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:26:46.332206 31799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:26:46.332504 31799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:26:46.333570 31799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:26:46.335021 31799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:26:46.335098 31799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:46.335218 31799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:26:46.335268 31799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:46.367419 31799 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.66:40749
I20260810 23:26:46.367466 31995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.66:40749 every 8 connection(s)
I20260810 23:26:46.368738 31799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
I20260810 23:26:46.375025 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 31799
I20260810 23:26:46.375229 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance
I20260810 23:26:46.375797 15037 external_mini_cluster.cc:1428] Setting key 897c1aaa165c6dbc4131b100314e0fa5
Aug 10 23:26:46 dist-test-slave-rmhg krb5kdc[31410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404406, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:26:46.395624 32000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:46.371988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:33295 (local address 127.14.175.66:55455)
0810 23:26:46.372780 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:46.372822 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:46.374617 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:46.375641 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:46.375654 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:46.376445 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:46.377378 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:46.377410 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:46.378882 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:46.378888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:46.380864 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:46.380883 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:46.382081 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:46.382088 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:46.382390 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:46.383139 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:46.383178 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:46.386244 (+ 3066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:46.390236 (+ 3992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:46.390247 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:46.390267 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:46.391260 (+ 993us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:46.391744 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:46.391749 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:46.391752 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:46.392021 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:46.392689 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:46.392699 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:46.393146 (+ 447us) client_negotiation.cc:769] Sending connection context
0810 23:26:46.393509 (+ 363us) client_negotiation.cc:240] Negotiation successful
0810 23:26:46.394112 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":276,"threads_started":1}
I20260810 23:26:46.396886 31999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:46.371456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:55455 (local address 127.14.175.126:33295)
0810 23:26:46.371997 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:46.372004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:46.372963 (+ 959us) server_negotiation.cc:407] Connection header received
0810 23:26:46.375010 (+ 2047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:46.375015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:46.375137 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:46.375261 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:46.377627 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:46.378673 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:46.381059 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:46.381654 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:46.381773 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:46.386593 (+ 4820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:46.386601 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:46.386605 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:46.386649 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:46.390034 (+ 3385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:46.391423 (+ 1389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:46.391430 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:46.391434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:46.391603 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:46.392155 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:46.392160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:46.392163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:46.392480 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:46.392618 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:46.396238 (+ 3620us) server_negotiation.cc:299] Negotiation successful
0810 23:26:46.396505 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":260,"threads_started":1}
I20260810 23:26:46.398353 31997 heartbeater.cc:347] Connected to a master server at 127.14.175.126:33295
I20260810 23:26:46.398780 31997 heartbeater.cc:464] Registering TS with master...
I20260810 23:26:46.399732 31997 heartbeater.cc:511] Master 127.14.175.126:33295 requested a full tablet report, sending...
I20260810 23:26:46.401643 31480 ts_manager.cc:194] Registered new tserver with Master: 9e2d6b54bccf4e94a495e31ae2e0307c (127.14.175.66:40749)
I20260810 23:26:46.402887 31480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.66@KRBTEST.COM'} at 127.14.175.66:55455
WARNING: no policy specified for kudu/127.14.175.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
I20260810 23:26:46.454675 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.67:0
--local_ip_for_outbound_sockets=127.14.175.67
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:33295
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:38061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:26:46.801550 32004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:26:46.801915 32004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:26:46.802039 32004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:26:46.809600 32004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:26:46.809731 32004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:26:46.809880 32004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.67
I20260810 23:26:46.820123 32004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:38061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:33295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.67
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:26:46.821506 32004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:26:46.822840 32004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:26:46.837785 32012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:46.837770 32017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:26:46.837816 32013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:26:46.838792 32004 server_base.cc:1034] running on GCE node
I20260810 23:26:46.839491 32004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:26:46.840602 32004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:26:46.841867 32004 hybrid_clock.cc:648] HybridClock initialized: now 1786404406841805 us; error 50 us; skew 500 ppm
I20260810 23:26:46.842267 32004 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:26:46 dist-test-slave-rmhg krb5kdc[31410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404406, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:26:46.851172 32004 init.cc:377] Logged in from keytab as kudu/127.14.175.67@KRBTEST.COM (short username kudu)
I20260810 23:26:46.852888 32004 webserver.cc:460] Webserver started at http://127.14.175.67:45031/ using document root <none> and password file <none>
I20260810 23:26:46.853763 32004 fs_manager.cc:359] Metadata directory not provided
I20260810 23:26:46.853899 32004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:26:46.854226 32004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:26:46.856856 31787 heartbeater.cc:503] Master 127.14.175.126:33295 was elected leader, sending a full tablet report...
I20260810 23:26:46.857640 32004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/data/instance:
uuid: "05ef6318a961473eb8188225248b6786"
format_stamp: "Formatted at 2026-08-10 23:26:46 on dist-test-slave-rmhg"
server_key: "3a084d1b78259a848130a8de0e53a5ea"
server_key_iv: "3931a68fcc71b4e9ea2216edae23e232"
server_key_version: "encryptionkey@0"
I20260810 23:26:46.858722 32004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance:
uuid: "05ef6318a961473eb8188225248b6786"
format_stamp: "Formatted at 2026-08-10 23:26:46 on dist-test-slave-rmhg"
server_key: "3a084d1b78259a848130a8de0e53a5ea"
server_key_iv: "3931a68fcc71b4e9ea2216edae23e232"
server_key_version: "encryptionkey@0"
I20260810 23:26:46.864786 32004 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260810 23:26:46.868621 32027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:46.870168 32004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:26:46.870345 32004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/wal
uuid: "05ef6318a961473eb8188225248b6786"
format_stamp: "Formatted at 2026-08-10 23:26:46 on dist-test-slave-rmhg"
server_key: "3a084d1b78259a848130a8de0e53a5ea"
server_key_iv: "3931a68fcc71b4e9ea2216edae23e232"
server_key_version: "encryptionkey@0"
I20260810 23:26:46.870548 32004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:26:46.891631 32004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:26:46.896234 32004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:26:46.896536 32004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:26:46.897574 32004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:26:46.899011 32004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:26:46.899092 32004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:46.899217 32004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:26:46.899272 32004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:26:46.930049 32004 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.67:35297
I20260810 23:26:46.930099 32215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.67:35297 every 8 connection(s)
I20260810 23:26:46.931370 32004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
I20260810 23:26:46.934144 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 32004
I20260810 23:26:46.934340 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance
I20260810 23:26:46.934864 15037 external_mini_cluster.cc:1428] Setting key 10226731520fb0aeab1a82f424798fc0
Aug 10 23:26:46 dist-test-slave-rmhg krb5kdc[31410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404406, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:26:46.952358 32221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:46.934084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:40219 (local address 127.14.175.126:33295)
0810 23:26:46.934794 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:46.934801 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:46.935693 (+ 892us) server_negotiation.cc:407] Connection header received
0810 23:26:46.937664 (+ 1971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:46.937670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:46.937769 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:46.937867 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:46.939604 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:46.940430 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:46.942145 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:46.942554 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:46.942691 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:46.946437 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:46.946443 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:46.946446 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:46.946523 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:46.948998 (+ 2475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:46.949822 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:46.949827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:46.949830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:46.949962 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:46.950421 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:46.950425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:46.950427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:46.950734 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:46.950876 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:46.951696 (+ 820us) server_negotiation.cc:299] Negotiation successful
0810 23:26:46.951950 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":415,"threads_started":1}
I20260810 23:26:46.953433 32223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:46.934626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:33295 (local address 127.14.175.67:40219)
0810 23:26:46.935529 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:46.935580 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:46.937371 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:46.938153 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:46.938163 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:46.938741 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:26:46.939405 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:46.939427 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:46.940590 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:46.940594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:46.941987 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:46.942002 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:46.942805 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:46.942813 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:46.943063 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:46.943787 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:26:46.943816 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:26:46.946184 (+ 2368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:26:46.949181 (+ 2997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:46.949192 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:46.949215 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:46.949680 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:46.950072 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:26:46.950077 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:26:46.950081 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:26:46.950300 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:26:46.950869 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:26:46.950878 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:26:46.951271 (+ 393us) client_negotiation.cc:769] Sending connection context
0810 23:26:46.951542 (+ 271us) client_negotiation.cc:240] Negotiation successful
0810 23:26:46.952159 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":334,"threads_started":1}
I20260810 23:26:46.955083 32217 heartbeater.cc:347] Connected to a master server at 127.14.175.126:33295
I20260810 23:26:46.955401 32217 heartbeater.cc:464] Registering TS with master...
I20260810 23:26:46.956230 32217 heartbeater.cc:511] Master 127.14.175.126:33295 requested a full tablet report, sending...
I20260810 23:26:46.957882 31480 ts_manager.cc:194] Registered new tserver with Master: 05ef6318a961473eb8188225248b6786 (127.14.175.67:35297)
I20260810 23:26:46.958700 31480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.67@KRBTEST.COM'} at 127.14.175.67:40219
I20260810 23:26:46.962759 15037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:26:46.980175 32221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:46.967458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52154 (local address 127.14.175.126:33295)
0810 23:26:46.967706 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:46.967713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:46.968144 (+ 431us) server_negotiation.cc:407] Connection header received
0810 23:26:46.968296 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:46.968301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:46.968388 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:46.968476 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:46.970023 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:46.970822 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:46.972126 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:46.972547 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:46.972655 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:46.975050 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:46.975056 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:46.975060 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:46.975096 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:46.977724 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:46.978376 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:46.978380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:46.978382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:46.978501 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:46.978782 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:46.978785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:46.978787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:46.979117 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:46.979300 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:46.979674 (+ 374us) server_negotiation.cc:299] Negotiation successful
0810 23:26:46.979869 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 23:26:46.984573 31480 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:26:46.991680 31480 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:26:47.017827 32240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:47.005877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38954 (local address 127.14.175.65:45495)
0810 23:26:47.006505 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:47.006514 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:47.008016 (+ 1502us) server_negotiation.cc:407] Connection header received
0810 23:26:47.008347 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:47.008356 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:47.008623 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:47.008819 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:26:47.009900 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:47.011510 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:47.014485 (+ 2975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:47.016059 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:47.016182 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:47.016428 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:47.017211 (+ 783us) server_negotiation.cc:299] Negotiation successful
0810 23:26:47.017398 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":308,"threads_started":1}
I20260810 23:26:47.019006 32237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:47.006659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:45495 (local address 127.0.0.1:38954)
0810 23:26:47.007892 (+ 1233us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:47.007909 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:47.008133 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:47.009133 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:47.009139 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:47.009220 (+ 81us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:26:47.009672 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:47.009686 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:47.011719 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:47.011724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:47.014302 (+ 2578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:47.014318 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:47.016698 (+ 2380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:47.016705 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:47.016890 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:47.016950 (+ 60us) client_negotiation.cc:769] Sending connection context
0810 23:26:47.018389 (+ 1439us) client_negotiation.cc:240] Negotiation successful
0810 23:26:47.018620 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1117,"thread_start_us":304,"threads_started":1}
I20260810 23:26:47.020844 32243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:47.004752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:40749 (local address 127.0.0.1:58262)
0810 23:26:47.007369 (+ 2617us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:47.007407 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:47.007598 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:47.010306 (+ 2708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:47.010311 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:47.010337 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:26:47.010783 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:47.010796 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:47.014422 (+ 3626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:47.014428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:47.016547 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:47.016583 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:47.020033 (+ 3450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:47.020040 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:47.020237 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:47.020277 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 23:26:47.020360 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 23:26:47.020504 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2354,"thread_start_us":2831,"threads_started":1}
I20260810 23:26:47.021445 32238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:47.004977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58262 (local address 127.14.175.66:40749)
0810 23:26:47.009649 (+ 4672us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:47.009658 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:47.009690 (+ 32us) server_negotiation.cc:407] Connection header received
0810 23:26:47.009787 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:47.009795 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:47.010065 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:47.011004 (+ 939us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:26:47.011264 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:47.014253 (+ 2989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:47.018225 (+ 3972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:47.019780 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:47.020598 (+ 818us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:47.020826 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:47.020903 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 23:26:47.021087 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4486,"thread_start_us":4350,"threads_started":1}
I20260810 23:26:47.023952 31681 tablet_service.cc:1467] Processing CreateTablet for tablet bd0ff9844f29491ca2a40fc25e9f5132 (DEFAULT_TABLE table=test-table [id=fb1a201ce82d4661a318ccba88a414c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:26:47.025373 31681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd0ff9844f29491ca2a40fc25e9f5132. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:26:47.025259 31895 tablet_service.cc:1467] Processing CreateTablet for tablet bd0ff9844f29491ca2a40fc25e9f5132 (DEFAULT_TABLE table=test-table [id=fb1a201ce82d4661a318ccba88a414c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:26:47.026551 31895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd0ff9844f29491ca2a40fc25e9f5132. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:26:47.028724 32246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:47.008334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54166 (local address 127.14.175.67:35297)
0810 23:26:47.019914 (+ 11580us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:47.019925 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:47.020890 (+ 965us) server_negotiation.cc:407] Connection header received
0810 23:26:47.021000 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:47.021007 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:47.021312 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:47.021521 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:26:47.022170 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:47.023761 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:47.026194 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:47.027586 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:47.027724 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:47.027959 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:47.028211 (+ 252us) server_negotiation.cc:299] Negotiation successful
0810 23:26:47.028392 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11363,"thread_start_us":11603,"threads_started":1}
I20260810 23:26:47.029610 32245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:47.007991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:35297 (local address 127.0.0.1:54166)
0810 23:26:47.020589 (+ 12598us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:47.020610 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:47.020750 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:47.021524 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:47.021529 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:47.021559 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:26:47.021912 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:47.021926 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:47.023925 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:47.023930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:47.025998 (+ 2068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:47.026014 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:47.027805 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:47.027812 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:47.027982 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:47.028019 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 23:26:47.028129 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 23:26:47.028268 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12461,"thread_start_us":12313,"threads_started":1}
I20260810 23:26:47.033871 32114 tablet_service.cc:1467] Processing CreateTablet for tablet bd0ff9844f29491ca2a40fc25e9f5132 (DEFAULT_TABLE table=test-table [id=fb1a201ce82d4661a318ccba88a414c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:26:47.035564 32114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd0ff9844f29491ca2a40fc25e9f5132. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:26:47.038738 32247 tablet_bootstrap.cc:492] T bd0ff9844f29491ca2a40fc25e9f5132 P 9e2d6b54bccf4e94a495e31ae2e0307c: Bootstrap starting.
I20260810 23:26:47.041934 32248 tablet_bootstrap.cc:492] T bd0ff9844f29491ca2a40fc25e9f5132 P ae494906d89e46548c3c8545331c59a3: Bootstrap starting.
I20260810 23:26:47.042546 32247 tablet_bootstrap.cc:654] T bd0ff9844f29491ca2a40fc25e9f5132 P 9e2d6b54bccf4e94a495e31ae2e0307c: Neither blocks nor log segments found. Creating new log.
I20260810 23:26:47.044008 32247 log.cc:824] T bd0ff9844f29491ca2a40fc25e9f5132 P 9e2d6b54bccf4e94a495e31ae2e0307c: Log is configured to *not* fsync() on all Append() calls
I20260810 23:26:47.045979 32248 tablet_bootstrap.cc:654] T bd0ff9844f29491ca2a40fc25e9f5132 P ae494906d89e46548c3c8545331c59a3: Neither blocks nor log segments found. Creating new log.
I20260810 23:26:47.047394 32248 log.cc:824] T bd0ff9844f29491ca2a40fc25e9f5132 P ae494906d89e46548c3c8545331c59a3: Log is configured to *not* fsync() on all Append() calls
I20260810 23:26:47.047776 32247 tablet_bootstrap.cc:492] T bd0ff9844f29491ca2a40fc25e9f5132 P 9e2d6b54bccf4e94a495e31ae2e0307c: No bootstrap required, opened a new log
I20260810 23:26:47.048105 32247 ts_tablet_manager.cc:1397] T bd0ff9844f29491ca2a40fc25e9f5132 P 9e2d6b54bccf4e94a495e31ae2e0307c: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260810 23:26:47.050254 32248 tablet_bootstrap.cc:492] T bd0ff9844f29491ca2a40fc25e9f5132 P ae494906d89e46548c3c8545331c59a3: No bootstrap required, opened a new log
I20260810 23:26:47.050547 32248 ts_tablet_manager.cc:1397] T bd0ff9844f29491ca2a40fc25e9f5132 P ae494906d89e46548c3c8545331c59a3: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260810 23:26:47.051791 32254 tablet_bootstrap.cc:492] T bd0ff9844f29491ca2a40fc25e9f5132 P 05ef6318a961473eb8188225248b6786: Bootstrap starting.
I20260810 23:26:47.055732 32254 tablet_bootstrap.cc:654] T bd0ff9844f29491ca2a40fc25e9f5132 P 05ef6318a961473eb8188225248b6786: Neither blocks nor log segments found. Creating new log.
I20260810 23:26:47.057089 32254 log.cc:824] T bd0ff9844f29491ca2a40fc25e9f5132 P 05ef6318a961473eb8188225248b6786: Log is configured to *not* fsync() on all Append() calls
I20260810 23:26:47.057250 32247 raft_consensus.cc:348] T bd0ff9844f29491ca2a40fc25e9f5132 P 9e2d6b54bccf4e94a495e31ae2e0307c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2d6b54bccf4e94a495e31ae2e0307c" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 40749 } } peers { permanent_uuid: "05ef6318a961473eb8188225248b6786" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 35297 } } peers { permanent_uuid: "ae494906d89e46548c3c8545331c59a3" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 45495 } }
I20260810 23:26:47.057765 32247 raft_consensus.cc:374] T bd0ff9844f29491ca2a40fc25e9f5132 P 9e2d6b54bccf4e94a495e31ae2e0307c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:26:47.057870 32247 raft_consensus.cc:729] T bd0ff9844f29491ca2a40fc25e9f5132 P 9e2d6b54bccf4e94a495e31ae2e0307c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e2d6b54bccf4e94a495e31ae2e0307c, State: Initialized, Role: FOLLOWER
I20260810 23:26:47.058492 32247 consensus_queue.cc:260] T bd0ff9844f29491ca2a40fc25e9f5132 P 9e2d6b54bccf4e94a495e31ae2e0307c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2d6b54bccf4e94a495e31ae2e0307c" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 40749 } } peers { permanent_uuid: "05ef6318a961473eb8188225248b6786" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 35297 } } peers { permanent_uuid: "ae494906d89e46548c3c8545331c59a3" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 45495 } }
I20260810 23:26:47.059615 31997 heartbeater.cc:503] Master 127.14.175.126:33295 was elected leader, sending a full tablet report...
I20260810 23:26:47.060392 32247 ts_tablet_manager.cc:1428] T bd0ff9844f29491ca2a40fc25e9f5132 P 9e2d6b54bccf4e94a495e31ae2e0307c: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260810 23:26:47.061244 32248 raft_consensus.cc:348] T bd0ff9844f29491ca2a40fc25e9f5132 P ae494906d89e46548c3c8545331c59a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2d6b54bccf4e94a495e31ae2e0307c" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 40749 } } peers { permanent_uuid: "05ef6318a961473eb8188225248b6786" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 35297 } } peers { permanent_uuid: "ae494906d89e46548c3c8545331c59a3" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 45495 } }
I20260810 23:26:47.061817 32248 raft_consensus.cc:374] T bd0ff9844f29491ca2a40fc25e9f5132 P ae494906d89e46548c3c8545331c59a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:26:47.061928 32248 raft_consensus.cc:729] T bd0ff9844f29491ca2a40fc25e9f5132 P ae494906d89e46548c3c8545331c59a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae494906d89e46548c3c8545331c59a3, State: Initialized, Role: FOLLOWER
I20260810 23:26:47.062574 32248 consensus_queue.cc:260] T bd0ff9844f29491ca2a40fc25e9f5132 P ae494906d89e46548c3c8545331c59a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2d6b54bccf4e94a495e31ae2e0307c" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 40749 } } peers { permanent_uuid: "05ef6318a961473eb8188225248b6786" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 35297 } } peers { permanent_uuid: "ae494906d89e46548c3c8545331c59a3" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 45495 } }
I20260810 23:26:47.064496 32248 ts_tablet_manager.cc:1428] T bd0ff9844f29491ca2a40fc25e9f5132 P ae494906d89e46548c3c8545331c59a3: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260810 23:26:47.064795 32254 tablet_bootstrap.cc:492] T bd0ff9844f29491ca2a40fc25e9f5132 P 05ef6318a961473eb8188225248b6786: No bootstrap required, opened a new log
I20260810 23:26:47.065120 32254 ts_tablet_manager.cc:1397] T bd0ff9844f29491ca2a40fc25e9f5132 P 05ef6318a961473eb8188225248b6786: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260810 23:26:47.073660 32254 raft_consensus.cc:348] T bd0ff9844f29491ca2a40fc25e9f5132 P 05ef6318a961473eb8188225248b6786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2d6b54bccf4e94a495e31ae2e0307c" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 40749 } } peers { permanent_uuid: "05ef6318a961473eb8188225248b6786" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 35297 } } peers { permanent_uuid: "ae494906d89e46548c3c8545331c59a3" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 45495 } }
I20260810 23:26:47.074084 32254 raft_consensus.cc:374] T bd0ff9844f29491ca2a40fc25e9f5132 P 05ef6318a961473eb8188225248b6786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:26:47.074199 32254 raft_consensus.cc:729] T bd0ff9844f29491ca2a40fc25e9f5132 P 05ef6318a961473eb8188225248b6786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05ef6318a961473eb8188225248b6786, State: Initialized, Role: FOLLOWER
W20260810 23:26:47.074494 31788 tablet.cc:2369] T bd0ff9844f29491ca2a40fc25e9f5132 P ae494906d89e46548c3c8545331c59a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:26:47.074827 32254 consensus_queue.cc:260] T bd0ff9844f29491ca2a40fc25e9f5132 P 05ef6318a961473eb8188225248b6786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2d6b54bccf4e94a495e31ae2e0307c" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 40749 } } peers { permanent_uuid: "05ef6318a961473eb8188225248b6786" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 35297 } } peers { permanent_uuid: "ae494906d89e46548c3c8545331c59a3" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 45495 } }
I20260810 23:26:47.075994 32217 heartbeater.cc:503] Master 127.14.175.126:33295 was elected leader, sending a full tablet report...
I20260810 23:26:47.076562 32254 ts_tablet_manager.cc:1428] T bd0ff9844f29491ca2a40fc25e9f5132 P 05ef6318a961473eb8188225248b6786: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
W20260810 23:26:47.120886 31998 tablet.cc:2369] T bd0ff9844f29491ca2a40fc25e9f5132 P 9e2d6b54bccf4e94a495e31ae2e0307c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:26:47.154156 32259 raft_consensus.cc:482] T bd0ff9844f29491ca2a40fc25e9f5132 P 9e2d6b54bccf4e94a495e31ae2e0307c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:26:47.154466 32259 raft_consensus.cc:504] T bd0ff9844f29491ca2a40fc25e9f5132 P 9e2d6b54bccf4e94a495e31ae2e0307c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2d6b54bccf4e94a495e31ae2e0307c" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 40749 } } peers { permanent_uuid: "05ef6318a961473eb8188225248b6786" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 35297 } } peers { permanent_uuid: "ae494906d89e46548c3c8545331c59a3" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 45495 } }
I20260810 23:26:47.156319 32259 leader_election.cc:288] T bd0ff9844f29491ca2a40fc25e9f5132 P 9e2d6b54bccf4e94a495e31ae2e0307c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05ef6318a961473eb8188225248b6786 (127.14.175.67:35297), ae494906d89e46548c3c8545331c59a3 (127.14.175.65:45495)
I20260810 23:26:47.165222 32240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:47.156787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:48711 (local address 127.14.175.65:45495)
0810 23:26:47.157007 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:47.157014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:47.158038 (+ 1024us) server_negotiation.cc:407] Connection header received
0810 23:26:47.158336 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:47.158341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:47.158420 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:47.158518 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:26:47.159316 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:47.160394 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:47.163216 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:47.164088 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:47.164283 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:47.164369 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:47.164747 (+ 378us) server_negotiation.cc:299] Negotiation successful
0810 23:26:47.164928 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 23:26:47.165551 32246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:47.157297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:48051 (local address 127.14.175.67:35297)
0810 23:26:47.157552 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:47.157558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:47.157705 (+ 147us) server_negotiation.cc:407] Connection header received
0810 23:26:47.158042 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:47.158049 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:47.158117 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:47.158170 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:26:47.159054 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:47.160114 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:47.163196 (+ 3082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:47.164548 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:47.164718 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:47.164789 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:47.165115 (+ 326us) server_negotiation.cc:299] Negotiation successful
0810 23:26:47.165295 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260810 23:26:47.165694 32266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:47.156751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:35297 (local address 127.14.175.66:48051)
0810 23:26:47.157582 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:47.157602 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:47.157845 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:47.158367 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:47.158371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:47.158396 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:26:47.158886 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:47.158897 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:47.160286 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:47.160291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:47.162967 (+ 2676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:47.162987 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:47.164772 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:47.164778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:47.164939 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:47.164969 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 23:26:47.165053 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 23:26:47.165232 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"mutex_wait_us":103,"thread_start_us":470,"threads_started":1}
I20260810 23:26:47.166225 32265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:47.156803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:45495 (local address 127.14.175.66:48711)
0810 23:26:47.157902 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0810 23:26:47.157920 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:26:47.158159 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:26:47.158556 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:26:47.158559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:26:47.158590 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:26:47.159135 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:47.159147 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:47.160659 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:47.160664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:47.162968 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:26:47.162987 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:26:47.164340 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:26:47.164346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:26:47.164539 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:47.164568 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 23:26:47.165694 (+ 1126us) client_negotiation.cc:240] Negotiation successful
0810 23:26:47.165836 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"mutex_wait_us":77,"thread_start_us":1024,"threads_started":1}
I20260810 23:26:47.167057 32145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd0ff9844f29491ca2a40fc25e9f5132" candidate_uuid: "9e2d6b54bccf4e94a495e31ae2e0307c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ef6318a961473eb8188225248b6786" is_pre_election: true
I20260810 23:26:47.167285 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd0ff9844f29491ca2a40fc25e9f5132" candidate_uuid: "9e2d6b54bccf4e94a495e31ae2e0307c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae494906d89e46548c3c8545331c59a3" is_pre_election: true
I20260810 23:26:47.167757 32145 raft_consensus.cc:2454] T bd0ff9844f29491ca2a40fc25e9f5132 P 05ef6318a961473eb8188225248b6786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e2d6b54bccf4e94a495e31ae2e0307c in term 0.
I20260810 23:26:47.167809 31711 raft_consensus.cc:2454] T bd0ff9844f29491ca2a40fc25e9f5132 P ae494906d89e46548c3c8545331c59a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e2d6b54bccf4e94a495e31ae2e0307c in term 0.
I20260810 23:26:47.168527 31825 leader_election.cc:302] T bd0ff9844f29491ca2a40fc25e9f5132 P 9e2d6b54bccf4e94a495e31ae2e0307c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e2d6b54bccf4e94a495e31ae2e0307c, ae494906d89e46548c3c8545331c59a3; no voters:
I20260810 23:26:47.168944 32259 raft_consensus.cc:2781] T bd0ff9844f29491ca2a40fc25e9f5132 P 9e2d6b54bccf4e94a495e31ae2e0307c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:26:47.169109 32259 raft_consensus.cc:482] T bd0ff9844f29491ca2a40fc25e9f5132 P 9e2d6b54bccf4e94a495e31ae2e0307c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:26:47.169246 32259 raft_consensus.cc:3037] T bd0ff9844f29491ca2a40fc25e9f5132 P 9e2d6b54bccf4e94a495e31ae2e0307c [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:26:47.172472 32259 raft_consensus.cc:504] T bd0ff9844f29491ca2a40fc25e9f5132 P 9e2d6b54bccf4e94a495e31ae2e0307c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2d6b54bccf4e94a495e31ae2e0307c" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 40749 } } peers { permanent_uuid: "05ef6318a961473eb8188225248b6786" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 35297 } } peers { permanent_uuid: "ae494906d89e46548c3c8545331c59a3" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 45495 } }
I20260810 23:26:47.173309 32259 leader_election.cc:288] T bd0ff9844f29491ca2a40fc25e9f5132 P 9e2d6b54bccf4e94a495e31ae2e0307c [CANDIDATE]: Term 1 election: Requested vote from peers 05ef6318a961473eb8188225248b6786 (127.14.175.67:35297), ae494906d89e46548c3c8545331c59a3 (127.14.175.65:45495)
I20260810 23:26:47.174067 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd0ff9844f29491ca2a40fc25e9f5132" candidate_uuid: "9e2d6b54bccf4e94a495e31ae2e0307c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae494906d89e46548c3c8545331c59a3"
I20260810 23:26:47.174093 32145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd0ff9844f29491ca2a40fc25e9f5132" candidate_uuid: "9e2d6b54bccf4e94a495e31ae2e0307c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ef6318a961473eb8188225248b6786"
I20260810 23:26:47.174278 31711 raft_consensus.cc:3037] T bd0ff9844f29491ca2a40fc25e9f5132 P ae494906d89e46548c3c8545331c59a3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:26:47.174325 32145 raft_consensus.cc:3037] T bd0ff9844f29491ca2a40fc25e9f5132 P 05ef6318a961473eb8188225248b6786 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:26:47.176623 32145 raft_consensus.cc:2454] T bd0ff9844f29491ca2a40fc25e9f5132 P 05ef6318a961473eb8188225248b6786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e2d6b54bccf4e94a495e31ae2e0307c in term 1.
I20260810 23:26:47.176720 31711 raft_consensus.cc:2454] T bd0ff9844f29491ca2a40fc25e9f5132 P ae494906d89e46548c3c8545331c59a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e2d6b54bccf4e94a495e31ae2e0307c in term 1.
I20260810 23:26:47.177094 31824 leader_election.cc:302] T bd0ff9844f29491ca2a40fc25e9f5132 P 9e2d6b54bccf4e94a495e31ae2e0307c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05ef6318a961473eb8188225248b6786, 9e2d6b54bccf4e94a495e31ae2e0307c; no voters:
I20260810 23:26:47.177445 32259 raft_consensus.cc:2781] T bd0ff9844f29491ca2a40fc25e9f5132 P 9e2d6b54bccf4e94a495e31ae2e0307c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:26:47.177882 32259 raft_consensus.cc:686] T bd0ff9844f29491ca2a40fc25e9f5132 P 9e2d6b54bccf4e94a495e31ae2e0307c [term 1 LEADER]: Becoming Leader. State: Replica: 9e2d6b54bccf4e94a495e31ae2e0307c, State: Running, Role: LEADER
I20260810 23:26:47.178368 32259 consensus_queue.cc:237] T bd0ff9844f29491ca2a40fc25e9f5132 P 9e2d6b54bccf4e94a495e31ae2e0307c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2d6b54bccf4e94a495e31ae2e0307c" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 40749 } } peers { permanent_uuid: "05ef6318a961473eb8188225248b6786" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 35297 } } peers { permanent_uuid: "ae494906d89e46548c3c8545331c59a3" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 45495 } }
W20260810 23:26:47.183003 32218 tablet.cc:2369] T bd0ff9844f29491ca2a40fc25e9f5132 P 05ef6318a961473eb8188225248b6786: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:26:47.183169 31479 catalog_manager.cc:5495] T bd0ff9844f29491ca2a40fc25e9f5132 P 9e2d6b54bccf4e94a495e31ae2e0307c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e2d6b54bccf4e94a495e31ae2e0307c (127.14.175.66). New cstate: current_term: 1 leader_uuid: "9e2d6b54bccf4e94a495e31ae2e0307c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2d6b54bccf4e94a495e31ae2e0307c" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 40749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ef6318a961473eb8188225248b6786" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 35297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae494906d89e46548c3c8545331c59a3" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 45495 } health_report { overall_health: UNKNOWN } } }
I20260810 23:26:47.211174 32238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:47.205638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58272 (local address 127.14.175.66:40749)
0810 23:26:47.205875 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:47.205881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:47.205902 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:26:47.206018 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:47.206039 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:47.206135 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:47.206283 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:26:47.206687 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:47.207411 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:47.208801 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:47.209270 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:47.209388 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:47.209903 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:26:47.210541 (+ 638us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:26:47.210688 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:47.210791 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 23:26:47.210886 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 23:26:47.220310 31711 raft_consensus.cc:1261] T bd0ff9844f29491ca2a40fc25e9f5132 P ae494906d89e46548c3c8545331c59a3 [term 1 FOLLOWER]: Refusing update from remote peer 9e2d6b54bccf4e94a495e31ae2e0307c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:26:47.220315 32145 raft_consensus.cc:1261] T bd0ff9844f29491ca2a40fc25e9f5132 P 05ef6318a961473eb8188225248b6786 [term 1 FOLLOWER]: Refusing update from remote peer 9e2d6b54bccf4e94a495e31ae2e0307c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:26:47.221442 32259 consensus_queue.cc:1035] T bd0ff9844f29491ca2a40fc25e9f5132 P 9e2d6b54bccf4e94a495e31ae2e0307c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae494906d89e46548c3c8545331c59a3" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:26:47.221767 32268 consensus_queue.cc:1035] T bd0ff9844f29491ca2a40fc25e9f5132 P 9e2d6b54bccf4e94a495e31ae2e0307c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ef6318a961473eb8188225248b6786" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:26:47.232259 32273 mvcc.cc:205] Tried to move back new op lower bound from 7317112451963871232 to 7317112451806625792. Current Snapshot: MvccSnapshot[applied={T|T < 7317112451963871232}]
I20260810 23:26:47.251209 31479 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:26:47.251595 31479 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:26:47.255579 31479 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4bda6b8aac404bd3898b9fe4fbd12a82: Sending DeleteTablet for 3 replicas of tablet bd0ff9844f29491ca2a40fc25e9f5132
I20260810 23:26:47.256956 31895 tablet_service.cc:1514] Processing DeleteTablet for tablet bd0ff9844f29491ca2a40fc25e9f5132 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:26:47 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:58262
I20260810 23:26:47.257238 32114 tablet_service.cc:1514] Processing DeleteTablet for tablet bd0ff9844f29491ca2a40fc25e9f5132 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:26:47 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:54166
I20260810 23:26:47.257793 31681 tablet_service.cc:1514] Processing DeleteTablet for tablet bd0ff9844f29491ca2a40fc25e9f5132 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:26:47 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:38954
I20260810 23:26:47.260010 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 31581
I20260810 23:26:47.264171 32294 tablet_replica.cc:330] T bd0ff9844f29491ca2a40fc25e9f5132 P 05ef6318a961473eb8188225248b6786: stopping tablet replica
I20260810 23:26:47.264415 32295 tablet_replica.cc:330] T bd0ff9844f29491ca2a40fc25e9f5132 P ae494906d89e46548c3c8545331c59a3: stopping tablet replica
I20260810 23:26:47.264858 32293 tablet_replica.cc:330] T bd0ff9844f29491ca2a40fc25e9f5132 P 9e2d6b54bccf4e94a495e31ae2e0307c: stopping tablet replica
I20260810 23:26:47.264998 32295 raft_consensus.cc:2229] T bd0ff9844f29491ca2a40fc25e9f5132 P ae494906d89e46548c3c8545331c59a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:26:47.265336 32293 raft_consensus.cc:2229] T bd0ff9844f29491ca2a40fc25e9f5132 P 9e2d6b54bccf4e94a495e31ae2e0307c [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:26:47.265663 32295 raft_consensus.cc:2258] T bd0ff9844f29491ca2a40fc25e9f5132 P ae494906d89e46548c3c8545331c59a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:26:47.266026 32293 raft_consensus.cc:2258] T bd0ff9844f29491ca2a40fc25e9f5132 P 9e2d6b54bccf4e94a495e31ae2e0307c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:26:47.266433 32294 raft_consensus.cc:2229] T bd0ff9844f29491ca2a40fc25e9f5132 P 05ef6318a961473eb8188225248b6786 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:26:47.266987 32294 raft_consensus.cc:2258] T bd0ff9844f29491ca2a40fc25e9f5132 P 05ef6318a961473eb8188225248b6786 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:26:47.268587 32293 ts_tablet_manager.cc:1905] T bd0ff9844f29491ca2a40fc25e9f5132 P 9e2d6b54bccf4e94a495e31ae2e0307c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:26:47.268699 32295 ts_tablet_manager.cc:1905] T bd0ff9844f29491ca2a40fc25e9f5132 P ae494906d89e46548c3c8545331c59a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:26:47.269090 32294 ts_tablet_manager.cc:1905] T bd0ff9844f29491ca2a40fc25e9f5132 P 05ef6318a961473eb8188225248b6786: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:26:47.276048 32294 ts_tablet_manager.cc:1918] T bd0ff9844f29491ca2a40fc25e9f5132 P 05ef6318a961473eb8188225248b6786: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:26:47.276278 32293 ts_tablet_manager.cc:1918] T bd0ff9844f29491ca2a40fc25e9f5132 P 9e2d6b54bccf4e94a495e31ae2e0307c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:26:47.276346 32294 log.cc:1196] T bd0ff9844f29491ca2a40fc25e9f5132 P 05ef6318a961473eb8188225248b6786: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-2/wal/wals/bd0ff9844f29491ca2a40fc25e9f5132
I20260810 23:26:47.276593 32293 log.cc:1196] T bd0ff9844f29491ca2a40fc25e9f5132 P 9e2d6b54bccf4e94a495e31ae2e0307c: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-1/wal/wals/bd0ff9844f29491ca2a40fc25e9f5132
I20260810 23:26:47.276911 32294 ts_tablet_manager.cc:1939] T bd0ff9844f29491ca2a40fc25e9f5132 P 05ef6318a961473eb8188225248b6786: Deleting consensus metadata
I20260810 23:26:47.277292 32293 ts_tablet_manager.cc:1939] T bd0ff9844f29491ca2a40fc25e9f5132 P 9e2d6b54bccf4e94a495e31ae2e0307c: Deleting consensus metadata
I20260810 23:26:47.278110 32295 ts_tablet_manager.cc:1918] T bd0ff9844f29491ca2a40fc25e9f5132 P ae494906d89e46548c3c8545331c59a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:26:47.279096 32295 log.cc:1196] T bd0ff9844f29491ca2a40fc25e9f5132 P ae494906d89e46548c3c8545331c59a3: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404299514076-15037-0/minicluster-data/ts-0/wal/wals/bd0ff9844f29491ca2a40fc25e9f5132
I20260810 23:26:47.279119 31464 catalog_manager.cc:4841] TS 05ef6318a961473eb8188225248b6786 (127.14.175.67:35297): tablet bd0ff9844f29491ca2a40fc25e9f5132 (table test-table [id=fb1a201ce82d4661a318ccba88a414c0]) successfully deleted
I20260810 23:26:47.279505 31464 catalog_manager.cc:4841] TS 9e2d6b54bccf4e94a495e31ae2e0307c (127.14.175.66:40749): tablet bd0ff9844f29491ca2a40fc25e9f5132 (table test-table [id=fb1a201ce82d4661a318ccba88a414c0]) successfully deleted
I20260810 23:26:47.280148 32295 ts_tablet_manager.cc:1939] T bd0ff9844f29491ca2a40fc25e9f5132 P ae494906d89e46548c3c8545331c59a3: Deleting consensus metadata
I20260810 23:26:47.282377 31465 catalog_manager.cc:4841] TS ae494906d89e46548c3c8545331c59a3 (127.14.175.65:45495): tablet bd0ff9844f29491ca2a40fc25e9f5132 (table test-table [id=fb1a201ce82d4661a318ccba88a414c0]) successfully deleted
Aug 10 23:26:47 dist-test-slave-rmhg krb5kdc[31410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.65@KRBTEST.COM
I20260810 23:26:47.294899 32240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:47.261905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38970 (local address 127.14.175.65:45495)
0810 23:26:47.262112 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:47.262119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:47.279548 (+ 17429us) server_negotiation.cc:407] Connection header received
0810 23:26:47.279916 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:47.279921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:47.279990 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:47.280076 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:47.281437 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:47.282298 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:47.283589 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:47.284004 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:47.284123 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:47.288605 (+ 4482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:47.288625 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:47.288643 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:47.288695 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:47.291621 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:47.292484 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:47.292493 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:47.292495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:47.292657 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:47.293086 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:47.293090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:47.293092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:47.293614 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:47.293944 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:47.294297 (+ 353us) server_negotiation.cc:299] Negotiation successful
0810 23:26:47.294537 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 23:26:47.295548 31777 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:26:47.398530 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 31581
I20260810 23:26:47.422677 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 31799
Aug 10 23:26:47 dist-test-slave-rmhg krb5kdc[31410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.66@KRBTEST.COM
I20260810 23:26:47.439088 32238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:47.423434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58276 (local address 127.14.175.66:40749)
0810 23:26:47.423618 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:47.423625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:47.423699 (+ 74us) server_negotiation.cc:407] Connection header received
0810 23:26:47.423896 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:47.423900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:47.423990 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:47.424077 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:47.425416 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:47.426265 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:47.427514 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:47.427971 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:47.428083 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:47.432805 (+ 4722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:47.432820 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:47.432838 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:47.432892 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:47.436106 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:47.436857 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:47.436866 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:47.436868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:47.437027 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:47.437466 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:47.437470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:47.437472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:47.437877 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:47.438227 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:47.438532 (+ 305us) server_negotiation.cc:299] Negotiation successful
0810 23:26:47.438748 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 23:26:47.439972 31984 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:26:47.555538 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 31799
I20260810 23:26:47.581354 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 32004
Aug 10 23:26:47 dist-test-slave-rmhg krb5kdc[31410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.67@KRBTEST.COM
I20260810 23:26:47.597141 32246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:26:47.582146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54182 (local address 127.14.175.67:35297)
0810 23:26:47.582327 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 23:26:47.582333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:26:47.582354 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:26:47.582556 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:26:47.582560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:26:47.582626 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:26:47.582734 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:26:47.584106 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:47.584879 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:47.586241 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:26:47.586646 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:26:47.586766 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:26:47.591245 (+ 4479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:26:47.591263 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:26:47.591282 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:26:47.591325 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:26:47.594300 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:47.595011 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:47.595020 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:47.595023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:47.595188 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:26:47.595574 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:26:47.595578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:26:47.595580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:26:47.595961 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:26:47.596261 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0810 23:26:47.596584 (+ 323us) server_negotiation.cc:299] Negotiation successful
0810 23:26:47.596801 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 23:26:47.597798 32208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:26:47.686008 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 32004
I20260810 23:26:47.707628 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 31437
I20260810 23:26:47.708885 31534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:26:47.820518 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 31437
2026-08-10T23:26:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4697 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:26:47 dist-test-slave-rmhg krb5kdc[32308](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:26:47 dist-test-slave-rmhg krb5kdc[32308](info): set up 2 sockets
Aug 10 23:26:47 dist-test-slave-rmhg krb5kdc[32308](info): commencing operation
krb5kdc: starting...
W20260810 23:26:49.080637 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.207s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:26:49 dist-test-slave-rmhg krb5kdc[32308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:26:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:26:49Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.175.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.175.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.175.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/rangeradmin_127.14.175.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.175.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/rangeradmin_127.14.175.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.175.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.175.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.175.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/rangerlookup_127.14.175.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.175.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/rangerlookup_127.14.175.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.175.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.175.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.175.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/HTTP_127.14.175.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.175.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/HTTP_127.14.175.84@KRBTEST.COM.keytab.
I20260810 23:26:49.490624 15037 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskwnDIbQ/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/postgres -l logfile start
2026-08-10 23:26:52.058 UTC [32350] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 23:26:52.058 UTC [32350] LOG: listening on IPv4 address "127.14.175.84", port 35319
2026-08-10 23:26:52.062 UTC [32350] LOG: listening on Unix socket "/tmp/.s.PGSQL.35319"
2026-08-10 23:26:52.086 UTC [32353] LOG: database system was shut down at 2026-08-10 23:26:50 UTC
2026-08-10 23:26:52.091 UTC [32350] LOG: database system is ready to accept connections
I20260810 23:26:53.199970 15037 mini_postgres.cc:96] Postgres bound to 35319
2026-08-10 23:26:53.225 UTC [32369] FATAL: database "slave" does not exist
127.14.175.84:35319 - accepting connections
I20260810 23:26:53.226048 15037 mini_ranger.cc:161] Starting Ranger...
I20260810 23:26:53.261106 15037 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 23:26:53.748932 15037 mini_ranger.cc:87] Created ranger Postgres database
I20260810 23:26:53.749110 15037 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ranger-admin
2026-08-10 23:26:54,102 [I] DB FLAVOR :POSTGRES
2026-08-10 23:26:54,102 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 23:26:54,103 [I] Checking connection..
2026-08-10 23:26:54,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:35319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 23:26:54,385 [I] Checking connection passed.
2026-08-10 23:26:54,385 [I] --------- Verifying version history table ---------
2026-08-10 23:26:54,385 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:35319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:26:54,660 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 23:26:54,660 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:35319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:26:54,941 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 23:26:54,941 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 23:26:54,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:35319/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 23:26:55.198 UTC [32473] WARNING: there is no transaction in progress
2026-08-10 23:26:55,205 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 23:26:55,206 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:35319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:26:55,459 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 23:26:55,460 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 23:26:55,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:35319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 23:26:55,722 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:35319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 23:26:55,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:35319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rmhg.c.gcp-upstream.internal','N') ;"
2026-08-10 23:26:56,232 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:35319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 23:26:56,501 [I] Table x_portal_user does not exist in database ranger
2026-08-10 23:26:56,501 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:35319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 23:26:56,763 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 23:26:56,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:35319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 23:26:57,034 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:35319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 23:26:57,298 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 23:26:57,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:35319/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 23:26:58.302 UTC [32652] WARNING: there is no transaction in progress
2026-08-10 23:26:58.319 UTC [32652] WARNING: there is no transaction in progress
2026-08-10 23:26:58.336 UTC [32652] WARNING: there is no transaction in progress
2026-08-10 23:26:58.353 UTC [32652] WARNING: there is no transaction in progress
2026-08-10 23:26:58.369 UTC [32652] WARNING: there is no transaction in progress
2026-08-10 23:26:58.397 UTC [32652] WARNING: there is no transaction in progress
2026-08-10 23:26:58.457 UTC [32652] WARNING: there is no transaction in progress
2026-08-10 23:26:58.464 UTC [32652] WARNING: there is no transaction in progress
2026-08-10 23:26:58.479 UTC [32652] WARNING: there is no transaction in progress
2026-08-10 23:26:58.489 UTC [32652] WARNING: there is no transaction in progress
2026-08-10 23:26:58.501 UTC [32652] WARNING: there is no transaction in progress
2026-08-10 23:26:58.516 UTC [32652] WARNING: there is no transaction in progress
2026-08-10 23:26:58.526 UTC [32652] WARNING: there is no transaction in progress
2026-08-10 23:26:58.533 UTC [32652] WARNING: there is no transaction in progress
2026-08-10 23:26:58.541 UTC [32652] WARNING: there is no transaction in progress
2026-08-10 23:26:58.548 UTC [32652] WARNING: there is no transaction in progress
2026-08-10 23:26:59,212 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 23:26:59,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:35319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 23:26:59,476 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 23:26:59,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:35319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 23:26:59,747 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 23:26:59,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:35319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 23:27:00,003 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 23:27:00,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:35319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 23:27:00,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:35319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 23:27:00,544 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:35319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rmhg.c.gcp-upstream.internal';"
2026-08-10 23:27:00,808 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 23:27:00,808 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:35319/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 23:27:01,068 [I] DB_PATCHES have already been applied
I20260810 23:27:01.075030 15037 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ranger-admin:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskwnDIbQ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 23:27:01.075150 15037 mini_ranger.cc:193] Using host: 127.14.175.84
I20260810 23:27:01.097003 15037 mini_ranger.cc:227] Ranger admin URL: http://127.14.175.84:58247
Aug 10, 2026 11:27:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 11:27:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 11:27:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 11:27:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 11:27:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 11:27:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 11:27:02 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 23:27:03.718933 15037 mini_ranger.cc:160] Time spent starting Ranger: real 10.493s user 0.000s sys 0.082s
Aug 10, 2026 11:27:23 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 11:27:23 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 11:27:23 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 11:27:23 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 11:27:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 11:27:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 11:27:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 11:27:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 11:27:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 11:27:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 11:27:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 11:27:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 11:27:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 11:27:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 11:27:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 11:27:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 11:27:23 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 11:27:24 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 23:27:25.544798 15037 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.14.175.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.14.175.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.14.175.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/rangerkms_127.14.175.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.14.175.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/rangerkms_127.14.175.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.175.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.14.175.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.175.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/HTTP_127.14.175.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.175.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/HTTP_127.14.175.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 23:27:25 dist-test-slave-rmhg krb5kdc[32308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404445, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 23:27:25.751542 15037 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 23:27:25.779731 15037 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 23:27:26.587257 15037 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 23:27:26.587409 15037 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ranger-kms
I20260810 23:27:26.587569 15037 mini_ranger_kms.cc:227] Using postgres at 127.14.175.84:35319
2026-08-10 23:27:27,075 [I] DB FLAVOR :POSTGRES
2026-08-10 23:27:27,075 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 23:27:27,076 [I] Checking connection
2026-08-10 23:27:27,076 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:35319/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 23:27:27,326 [I] connection success
2026-08-10 23:27:27,327 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 23:27:27,327 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 23:27:27,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:35319/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 23:27:27,575 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 23:27:27,575 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 23:27:27,575 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 23:27:27,575 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:35319/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 23:27:27,856 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 23:27:28.397452 15037 mini_ranger_kms.cc:310] Created kms service
I20260810 23:27:28.570067 15037 mini_ranger_kms.cc:326] Created kudu user
I20260810 23:27:28.615863 15037 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 23:27:28 dist-test-slave-rmhg krb5kdc[32308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404445, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.14.175.84@KRBTEST.COM
I20260810 23:27:28.998131 15037 mini_ranger_kms.cc:384] Added ranger policy
I20260810 23:27:28.999963 15037 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ranger-kms:/tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 23:27:29.000042 15037 mini_ranger_kms.cc:240] Using host: 127.14.175.84
I20260810 23:27:29.030458 15037 mini_ranger_kms.cc:276] Ranger KMS PID: 485
I20260810 23:27:29.030656 15037 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.14.175.84:56589
Aug 10, 2026 11:27:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 11:27:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 11:27:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 11:27:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 11:27:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 11:27:30 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 23:27:31 dist-test-slave-rmhg krb5kdc[32308](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786404451, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.175.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:27:31.618728 15037 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.867s user 0.117s sys 0.415s
I20260810 23:27:31.619207 15037 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 23:27:31.619459 15037 mini_ranger_kms.cc:401] 127.14.175.84:56589/kms/v1/keys
Aug 10 23:27:31 dist-test-slave-rmhg krb5kdc[32308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404451, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.175.84@KRBTEST.COM for HTTP/127.14.175.84@KRBTEST.COM
Aug 10 23:27:32 dist-test-slave-rmhg krb5kdc[32308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404451, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.175.84@KRBTEST.COM for HTTP/127.14.175.84@KRBTEST.COM
Aug 10 23:27:34 dist-test-slave-rmhg krb5kdc[32308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.14.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
I20260810 23:27:34.624169 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:46057
--webserver_interface=127.14.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:42227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:46057
--ranger_config_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.175.84:56589/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:27:35.247422 566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:27:35.247869 566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:27:35.248075 566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:27:35.258885 566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:27:35.259123 566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:27:35.259258 566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:27:35.259384 566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:27:35.259479 566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:27:35.275255 566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:42227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/master-0/wal
--ranger_kms_url=127.14.175.84:56589/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:46057
--ranger_config_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:46057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:27:35.276757 566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:27:35.278316 566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:27:35.295389 577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:27:35.300631 580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:27:35.300631 566 server_base.cc:1034] running on GCE node
W20260810 23:27:35.300885 578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:27:35.301968 566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:27:35.303768 566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:27:35.305097 566 hybrid_clock.cc:648] HybridClock initialized: now 1786404455304988 us; error 37 us; skew 500 ppm
I20260810 23:27:35.305544 566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:27:35 dist-test-slave-rmhg krb5kdc[32308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404455, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:27:35.310031 566 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:27:35.312249 566 webserver.cc:460] Webserver started at http://127.14.175.126:37407/ using document root <none> and password file <none>
I20260810 23:27:35.313098 566 fs_manager.cc:359] Metadata directory not provided
I20260810 23:27:35.313359 566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:27:35.313704 566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:27:35 dist-test-slave-rmhg krb5kdc[32308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404455, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for HTTP/127.14.175.84@KRBTEST.COM
I20260810 23:27:35.374084 566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "05b36ba2de9e4fe2a34a608f5eb42b9c"
format_stamp: "Formatted at 2026-08-10 23:27:35 on dist-test-slave-rmhg"
server_key: "40a6a7b03c664fec0cbfeb1f898639b8"
server_key_iv: "842506bfcfda0b99ddd1d7abe9365dd4"
server_key_version: "kuduclusterkey@0"
I20260810 23:27:35.375201 566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "05b36ba2de9e4fe2a34a608f5eb42b9c"
format_stamp: "Formatted at 2026-08-10 23:27:35 on dist-test-slave-rmhg"
server_key: "40a6a7b03c664fec0cbfeb1f898639b8"
server_key_iv: "842506bfcfda0b99ddd1d7abe9365dd4"
server_key_version: "kuduclusterkey@0"
I20260810 23:27:35.381680 566 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260810 23:27:35.405787 596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:27:35.407472 566 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260810 23:27:35.407696 566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "05b36ba2de9e4fe2a34a608f5eb42b9c"
format_stamp: "Formatted at 2026-08-10 23:27:35 on dist-test-slave-rmhg"
server_key: "40a6a7b03c664fec0cbfeb1f898639b8"
server_key_iv: "842506bfcfda0b99ddd1d7abe9365dd4"
server_key_version: "kuduclusterkey@0"
I20260810 23:27:35.407927 566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:27:35.440788 566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:27:35.446696 566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:27:35.447278 566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:27:35.491827 566 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:46057
I20260810 23:27:35.491909 683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:46057 every 8 connection(s)
I20260810 23:27:35.493794 566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:27:35.498615 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 566
I20260810 23:27:35.498910 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:27:35.499902 15037 external_mini_cluster.cc:1428] Setting key 6a8c8d9a164c65c62695c135a3ac1392
I20260810 23:27:35.500540 684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 23:27:35 dist-test-slave-rmhg krb5kdc[32308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:27:35.521184 684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05b36ba2de9e4fe2a34a608f5eb42b9c: Bootstrap starting.
I20260810 23:27:35.526285 684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05b36ba2de9e4fe2a34a608f5eb42b9c: Neither blocks nor log segments found. Creating new log.
I20260810 23:27:35.528730 684 log.cc:824] T 00000000000000000000000000000000 P 05b36ba2de9e4fe2a34a608f5eb42b9c: Log is configured to *not* fsync() on all Append() calls
I20260810 23:27:35.531765 689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:27:35.502484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56206 (local address 127.14.175.126:46057)
0810 23:27:35.504046 (+ 1562us) server_negotiation.cc:206] Beginning negotiation
0810 23:27:35.504059 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:27:35.504111 (+ 52us) server_negotiation.cc:407] Connection header received
0810 23:27:35.507508 (+ 3397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:27:35.507539 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:27:35.508038 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:27:35.508455 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:27:35.510410 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:27:35.512351 (+ 1941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:27:35.514263 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:27:35.515256 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:27:35.515409 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:27:35.520843 (+ 5434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:27:35.520855 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:27:35.520875 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:27:35.520925 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:27:35.525524 (+ 4599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:27:35.526617 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:27:35.526626 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:27:35.526635 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:27:35.526897 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:27:35.527470 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:27:35.527474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:27:35.527476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:27:35.528275 (+ 799us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:27:35.528552 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0810 23:27:35.529285 (+ 733us) server_negotiation.cc:299] Negotiation successful
0810 23:27:35.529814 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1287,"thread_start_us":989,"threads_started":1}
I20260810 23:27:35.533991 684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05b36ba2de9e4fe2a34a608f5eb42b9c: No bootstrap required, opened a new log
I20260810 23:27:35.546435 684 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05b36ba2de9e4fe2a34a608f5eb42b9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b36ba2de9e4fe2a34a608f5eb42b9c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46057 } }
I20260810 23:27:35.546937 684 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05b36ba2de9e4fe2a34a608f5eb42b9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:27:35.547115 684 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05b36ba2de9e4fe2a34a608f5eb42b9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b36ba2de9e4fe2a34a608f5eb42b9c, State: Initialized, Role: FOLLOWER
I20260810 23:27:35.548059 684 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05b36ba2de9e4fe2a34a608f5eb42b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b36ba2de9e4fe2a34a608f5eb42b9c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46057 } }
I20260810 23:27:35.548480 684 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05b36ba2de9e4fe2a34a608f5eb42b9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:27:35.548714 684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05b36ba2de9e4fe2a34a608f5eb42b9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:27:35.548976 684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05b36ba2de9e4fe2a34a608f5eb42b9c [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:27:35.552066 684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05b36ba2de9e4fe2a34a608f5eb42b9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b36ba2de9e4fe2a34a608f5eb42b9c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46057 } }
I20260810 23:27:35.552629 684 leader_election.cc:302] T 00000000000000000000000000000000 P 05b36ba2de9e4fe2a34a608f5eb42b9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05b36ba2de9e4fe2a34a608f5eb42b9c; no voters:
I20260810 23:27:35.553721 684 leader_election.cc:288] T 00000000000000000000000000000000 P 05b36ba2de9e4fe2a34a608f5eb42b9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:27:35.553856 693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05b36ba2de9e4fe2a34a608f5eb42b9c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:27:35.555619 693 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05b36ba2de9e4fe2a34a608f5eb42b9c [term 1 LEADER]: Becoming Leader. State: Replica: 05b36ba2de9e4fe2a34a608f5eb42b9c, State: Running, Role: LEADER
I20260810 23:27:35.556392 693 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05b36ba2de9e4fe2a34a608f5eb42b9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b36ba2de9e4fe2a34a608f5eb42b9c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46057 } }
I20260810 23:27:35.557253 684 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05b36ba2de9e4fe2a34a608f5eb42b9c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:27:35.561189 694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05b36ba2de9e4fe2a34a608f5eb42b9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05b36ba2de9e4fe2a34a608f5eb42b9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b36ba2de9e4fe2a34a608f5eb42b9c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46057 } } }
I20260810 23:27:35.561282 696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05b36ba2de9e4fe2a34a608f5eb42b9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05b36ba2de9e4fe2a34a608f5eb42b9c. Latest consensus state: current_term: 1 leader_uuid: "05b36ba2de9e4fe2a34a608f5eb42b9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b36ba2de9e4fe2a34a608f5eb42b9c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46057 } } }
I20260810 23:27:35.562129 696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05b36ba2de9e4fe2a34a608f5eb42b9c [sys.catalog]: This master's current role is: LEADER
I20260810 23:27:35.562323 694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05b36ba2de9e4fe2a34a608f5eb42b9c [sys.catalog]: This master's current role is: LEADER
I20260810 23:27:35.564189 713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:27:35.578094 713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:27:35.580997 684 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 23:27:35.604004 713 catalog_manager.cc:1347] Generated new cluster ID: 1857f9fa6343404d98a3a92ae3657953
I20260810 23:27:35.604286 713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:27:35.628983 713 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:27:35.631742 713 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:27:35.645087 713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05b36ba2de9e4fe2a34a608f5eb42b9c: Generated new TSK 0
I20260810 23:27:35.646507 713 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
I20260810 23:27:36.553574 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.175.84:56589/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.65:0
--local_ip_for_outbound_sockets=127.14.175.65
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:46057
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:42227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:27:36 dist-test-slave-rmhg krb5kdc[32308](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786404456, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 23:27:37.325856 765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:27:37.326392 765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:27:37.326679 765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:27:37.340921 765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:27:37.341192 765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:27:37.341461 765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.65
I20260810 23:27:37.358527 765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:42227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.14.175.84:56589/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:46057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.65
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:27:37.360284 765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:27:37.362035 765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:27:37.391258 784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:27:37.392552 781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:27:37.394969 787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:27:37.398713 765 server_base.cc:1034] running on GCE node
I20260810 23:27:37.399860 765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:27:37.401403 765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:27:37.402712 765 hybrid_clock.cc:648] HybridClock initialized: now 1786404457402587 us; error 92 us; skew 500 ppm
I20260810 23:27:37.403254 765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:27:37 dist-test-slave-rmhg krb5kdc[32308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404457, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:27:37.410513 765 init.cc:377] Logged in from keytab as kudu/127.14.175.65@KRBTEST.COM (short username kudu)
I20260810 23:27:37.425834 765 webserver.cc:460] Webserver started at http://127.14.175.65:46653/ using document root <none> and password file <none>
I20260810 23:27:37.427233 765 fs_manager.cc:359] Metadata directory not provided
I20260810 23:27:37.427558 765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:27:37.428108 765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:27:37 dist-test-slave-rmhg krb5kdc[32308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404457, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for HTTP/127.14.175.84@KRBTEST.COM
I20260810 23:27:37.453795 765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-0/data/instance:
uuid: "3a5269f2083a4b56a9b64a242e309006"
format_stamp: "Formatted at 2026-08-10 23:27:37 on dist-test-slave-rmhg"
server_key: "1cfc195768b89ea6c27d46c4149ce46e"
server_key_iv: "103456ec4f84e74fdac2de49b450b36b"
server_key_version: "kuduclusterkey@0"
I20260810 23:27:37.455237 765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance:
uuid: "3a5269f2083a4b56a9b64a242e309006"
format_stamp: "Formatted at 2026-08-10 23:27:37 on dist-test-slave-rmhg"
server_key: "1cfc195768b89ea6c27d46c4149ce46e"
server_key_iv: "103456ec4f84e74fdac2de49b450b36b"
server_key_version: "kuduclusterkey@0"
I20260810 23:27:37.472524 765 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.008s sys 0.008s
Aug 10 23:27:37 dist-test-slave-rmhg krb5kdc[32308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404456, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for HTTP/127.14.175.84@KRBTEST.COM
I20260810 23:27:37.530280 796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:27:37.533567 765 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.007s sys 0.006s
I20260810 23:27:37.533789 765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-0/wal
uuid: "3a5269f2083a4b56a9b64a242e309006"
format_stamp: "Formatted at 2026-08-10 23:27:37 on dist-test-slave-rmhg"
server_key: "1cfc195768b89ea6c27d46c4149ce46e"
server_key_iv: "103456ec4f84e74fdac2de49b450b36b"
server_key_version: "kuduclusterkey@0"
I20260810 23:27:37.534046 765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:27:37.573015 765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:27:37.581132 765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:27:37.581724 765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:27:37.584187 765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:27:37.587391 765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:27:37.587625 765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:27:37.587886 765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:27:37.588052 765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:27:37.740078 765 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.65:35165
I20260810 23:27:37.740378 983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.65:35165 every 8 connection(s)
I20260810 23:27:37.742271 765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
I20260810 23:27:37.743645 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 765
I20260810 23:27:37.743886 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance
I20260810 23:27:37.744964 15037 external_mini_cluster.cc:1428] Setting key 36d6337d4292b48ce8576cee3eb6ce44
Aug 10 23:27:37 dist-test-slave-rmhg krb5kdc[32308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404457, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.175.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.66@KRBTEST.COM" created.
I20260810 23:27:37.816184 990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:27:37.765865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:35707 (local address 127.14.175.126:46057)
0810 23:27:37.777893 (+ 12028us) server_negotiation.cc:206] Beginning negotiation
0810 23:27:37.777899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:27:37.780646 (+ 2747us) server_negotiation.cc:407] Connection header received
0810 23:27:37.784443 (+ 3797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:27:37.784451 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:27:37.784579 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:27:37.784669 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:27:37.787987 (+ 3318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:27:37.789366 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:27:37.792814 (+ 3448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:27:37.793808 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:27:37.793932 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:27:37.805932 (+ 12000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:27:37.805945 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:27:37.805952 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:27:37.806012 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:27:37.810700 (+ 4688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:27:37.812098 (+ 1398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:27:37.812105 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:27:37.812109 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:27:37.812579 (+ 470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:27:37.813475 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:27:37.813482 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:27:37.813485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:27:37.813996 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:27:37.814159 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 23:27:37.815409 (+ 1250us) server_negotiation.cc:299] Negotiation successful
0810 23:27:37.815689 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11893,"thread_start_us":11660,"threads_started":1}
I20260810 23:27:37.819006 991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:27:37.766781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46057 (local address 127.14.175.65:35707)
0810 23:27:37.780199 (+ 13418us) negotiation.cc:107] Waiting for socket to connect
0810 23:27:37.780269 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0810 23:27:37.783999 (+ 3730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:27:37.785403 (+ 1404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:27:37.785417 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:27:37.786403 (+ 986us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:27:37.787694 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:27:37.787730 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:27:37.789720 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:27:37.789726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:27:37.792558 (+ 2832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:27:37.792583 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:27:37.794579 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:27:37.794591 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:27:37.795104 (+ 513us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:27:37.796205 (+ 1101us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:27:37.796281 (+ 76us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:27:37.800321 (+ 4040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:27:37.810976 (+ 10655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:27:37.810990 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:27:37.811014 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:27:37.811871 (+ 857us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:27:37.812839 (+ 968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:27:37.812845 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:27:37.812848 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:27:37.813308 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:27:37.814305 (+ 997us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:27:37.814316 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:27:37.814873 (+ 557us) client_negotiation.cc:769] Sending connection context
0810 23:27:37.816484 (+ 1611us) client_negotiation.cc:240] Negotiation successful
0810 23:27:37.817224 (+ 740us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12886,"thread_start_us":11314,"threads_started":1}
I20260810 23:27:37.821787 985 heartbeater.cc:347] Connected to a master server at 127.14.175.126:46057
I20260810 23:27:37.822203 985 heartbeater.cc:464] Registering TS with master...
I20260810 23:27:37.823107 985 heartbeater.cc:511] Master 127.14.175.126:46057 requested a full tablet report, sending...
I20260810 23:27:37.826033 623 ts_manager.cc:194] Registered new tserver with Master: 3a5269f2083a4b56a9b64a242e309006 (127.14.175.65:35165)
I20260810 23:27:37.828471 623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.65@KRBTEST.COM'} at 127.14.175.65:35707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
W20260810 23:27:37.850452 15037 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.14.175.66: real 0.101s user 0.004s sys 0.022s
I20260810 23:27:37.851569 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.175.84:56589/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.66:0
--local_ip_for_outbound_sockets=127.14.175.66
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:46057
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:42227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:27:37 dist-test-slave-rmhg krb5kdc[32308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404456, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for HTTP/127.14.175.84@KRBTEST.COM
W20260810 23:27:38.647753 998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:27:38.648083 998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:27:38.648212 998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:27:38.655979 998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:27:38.656116 998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:27:38.656260 998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.66
I20260810 23:27:38.668702 998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:42227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.14.175.84:56589/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:46057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.66
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:27:38.670226 998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:27:38.671846 998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:27:38.690347 1021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:27:38.690388 1020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:27:38.692381 998 server_base.cc:1034] running on GCE node
W20260810 23:27:38.693430 1023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:27:38.694416 998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:27:38.695839 998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:27:38.697189 998 hybrid_clock.cc:648] HybridClock initialized: now 1786404458697033 us; error 62 us; skew 500 ppm
I20260810 23:27:38.697672 998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:27:38 dist-test-slave-rmhg krb5kdc[32308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404458, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:27:38.707329 998 init.cc:377] Logged in from keytab as kudu/127.14.175.66@KRBTEST.COM (short username kudu)
I20260810 23:27:38.710783 998 webserver.cc:460] Webserver started at http://127.14.175.66:44129/ using document root <none> and password file <none>
I20260810 23:27:38.712167 998 fs_manager.cc:359] Metadata directory not provided
I20260810 23:27:38.712430 998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:27:38.712924 998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:27:38 dist-test-slave-rmhg krb5kdc[32308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404458, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for HTTP/127.14.175.84@KRBTEST.COM
I20260810 23:27:38.735718 998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-1/data/instance:
uuid: "e99115f3ba134d469d53977cceaddb48"
format_stamp: "Formatted at 2026-08-10 23:27:38 on dist-test-slave-rmhg"
server_key: "838dbc7ef109bf6d968cfb45862fc6dc"
server_key_iv: "bc85458f93c2a719f098751b1c95bd1f"
server_key_version: "kuduclusterkey@0"
I20260810 23:27:38.737274 998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance:
uuid: "e99115f3ba134d469d53977cceaddb48"
format_stamp: "Formatted at 2026-08-10 23:27:38 on dist-test-slave-rmhg"
server_key: "838dbc7ef109bf6d968cfb45862fc6dc"
server_key_iv: "bc85458f93c2a719f098751b1c95bd1f"
server_key_version: "kuduclusterkey@0"
I20260810 23:27:38.746382 998 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.001s sys 0.006s
I20260810 23:27:38.767401 1035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:27:38.769587 998 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.001s
I20260810 23:27:38.769876 998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-1/wal
uuid: "e99115f3ba134d469d53977cceaddb48"
format_stamp: "Formatted at 2026-08-10 23:27:38 on dist-test-slave-rmhg"
server_key: "838dbc7ef109bf6d968cfb45862fc6dc"
server_key_iv: "bc85458f93c2a719f098751b1c95bd1f"
server_key_version: "kuduclusterkey@0"
I20260810 23:27:38.770234 998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:27:38.798343 998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:27:38.803632 998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:27:38.804032 998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:27:38.805579 998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:27:38.807889 998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:27:38.807984 998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:27:38.808117 998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:27:38.808187 998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:27:38.833019 985 heartbeater.cc:503] Master 127.14.175.126:46057 was elected leader, sending a full tablet report...
I20260810 23:27:38.904807 998 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.66:43781
I20260810 23:27:38.905124 1219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.66:43781 every 8 connection(s)
I20260810 23:27:38.906656 998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
I20260810 23:27:38.913345 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 998
I20260810 23:27:38.913549 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance
I20260810 23:27:38.914281 15037 external_mini_cluster.cc:1428] Setting key a9a79654db239547bca6d16fac05ecf6
Aug 10 23:27:38 dist-test-slave-rmhg krb5kdc[32308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404458, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:27:38.945691 1227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:27:38.911505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:48803 (local address 127.14.175.126:46057)
0810 23:27:38.912629 (+ 1124us) server_negotiation.cc:206] Beginning negotiation
0810 23:27:38.912636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:27:38.914981 (+ 2345us) server_negotiation.cc:407] Connection header received
0810 23:27:38.919070 (+ 4089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:27:38.919077 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:27:38.919170 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:27:38.919312 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:27:38.921983 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:27:38.923809 (+ 1826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:27:38.927060 (+ 3251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:27:38.927975 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:27:38.928123 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:27:38.934179 (+ 6056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:27:38.934188 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:27:38.934193 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:27:38.934283 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:27:38.939799 (+ 5516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:27:38.941348 (+ 1549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:27:38.941355 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:27:38.941358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:27:38.941640 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:27:38.942582 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:27:38.942588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:27:38.942591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:27:38.943295 (+ 704us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:27:38.943551 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0810 23:27:38.944741 (+ 1190us) server_negotiation.cc:299] Negotiation successful
0810 23:27:38.945029 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":896,"thread_start_us":689,"threads_started":1}
I20260810 23:27:38.947486 1235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:27:38.912607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46057 (local address 127.14.175.66:48803)
0810 23:27:38.914304 (+ 1697us) negotiation.cc:107] Waiting for socket to connect
0810 23:27:38.914364 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 23:27:38.918705 (+ 4341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:27:38.919718 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:27:38.919733 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:27:38.920689 (+ 956us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:27:38.921751 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:27:38.921782 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:27:38.924031 (+ 2249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:27:38.924036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:27:38.926862 (+ 2826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:27:38.926883 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:27:38.928388 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:27:38.928399 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:27:38.928924 (+ 525us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:27:38.929861 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:27:38.929887 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:27:38.933733 (+ 3846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:27:38.940041 (+ 6308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:27:38.940055 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:27:38.940080 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:27:38.941142 (+ 1062us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:27:38.941798 (+ 656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:27:38.941804 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:27:38.941808 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:27:38.942409 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:27:38.943467 (+ 1058us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:27:38.943477 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:27:38.944122 (+ 645us) client_negotiation.cc:769] Sending connection context
0810 23:27:38.944506 (+ 384us) client_negotiation.cc:240] Negotiation successful
0810 23:27:38.945437 (+ 931us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1321,"thread_start_us":865,"threads_started":1}
I20260810 23:27:38.949712 1221 heartbeater.cc:347] Connected to a master server at 127.14.175.126:46057
WARNING: no policy specified for kudu/127.14.175.67@KRBTEST.COM; defaulting to no policy
I20260810 23:27:38.950219 1221 heartbeater.cc:464] Registering TS with master...
I20260810 23:27:38.951480 1221 heartbeater.cc:511] Master 127.14.175.126:46057 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.67@KRBTEST.COM" created.
I20260810 23:27:38.953732 623 ts_manager.cc:194] Registered new tserver with Master: e99115f3ba134d469d53977cceaddb48 (127.14.175.66:43781)
I20260810 23:27:38.955215 623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.66@KRBTEST.COM'} at 127.14.175.66:48803
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
I20260810 23:27:38.989269 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.175.84:56589/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.67:0
--local_ip_for_outbound_sockets=127.14.175.67
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:46057
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:42227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:27:39.733668 1241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:27:39.734135 1241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:27:39.734479 1241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:27:39.747764 1241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:27:39.748009 1241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:27:39.748229 1241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.67
I20260810 23:27:39.762301 1241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:42227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.14.175.84:56589/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:46057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.67
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:27:39.763955 1241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:27:39.765663 1241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:27:39.785614 1249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:27:39.785578 1255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:27:39.786546 1257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:27:39.787806 1241 server_base.cc:1034] running on GCE node
I20260810 23:27:39.788738 1241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:27:39.790463 1241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:27:39.791741 1241 hybrid_clock.cc:648] HybridClock initialized: now 1786404459791655 us; error 61 us; skew 500 ppm
I20260810 23:27:39.792189 1241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:27:39 dist-test-slave-rmhg krb5kdc[32308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404459, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:27:39.797899 1241 init.cc:377] Logged in from keytab as kudu/127.14.175.67@KRBTEST.COM (short username kudu)
I20260810 23:27:39.800628 1241 webserver.cc:460] Webserver started at http://127.14.175.67:44625/ using document root <none> and password file <none>
I20260810 23:27:39.801867 1241 fs_manager.cc:359] Metadata directory not provided
I20260810 23:27:39.802134 1241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:27:39.802603 1241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:27:39 dist-test-slave-rmhg krb5kdc[32308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404459, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for HTTP/127.14.175.84@KRBTEST.COM
I20260810 23:27:39.823587 1241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-2/data/instance:
uuid: "8722e66e949142b182767752dbbcfc1f"
format_stamp: "Formatted at 2026-08-10 23:27:39 on dist-test-slave-rmhg"
server_key: "996616138c901835f08762f90c219cc4"
server_key_iv: "8540ddef5f0c0af450446952db654a22"
server_key_version: "kuduclusterkey@0"
I20260810 23:27:39.825301 1241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance:
uuid: "8722e66e949142b182767752dbbcfc1f"
format_stamp: "Formatted at 2026-08-10 23:27:39 on dist-test-slave-rmhg"
server_key: "996616138c901835f08762f90c219cc4"
server_key_iv: "8540ddef5f0c0af450446952db654a22"
server_key_version: "kuduclusterkey@0"
I20260810 23:27:39.834652 1241 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.001s sys 0.006s
I20260810 23:27:39.853719 1268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:27:39.855775 1241 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260810 23:27:39.856089 1241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-2/wal
uuid: "8722e66e949142b182767752dbbcfc1f"
format_stamp: "Formatted at 2026-08-10 23:27:39 on dist-test-slave-rmhg"
server_key: "996616138c901835f08762f90c219cc4"
server_key_iv: "8540ddef5f0c0af450446952db654a22"
server_key_version: "kuduclusterkey@0"
I20260810 23:27:39.856446 1241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:27:39.886224 1241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:27:39.892330 1241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:27:39.892940 1241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:27:39.894572 1241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:27:39.897040 1241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:27:39.897135 1241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:27:39.897380 1241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:27:39.897544 1241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:27:39.960017 1221 heartbeater.cc:503] Master 127.14.175.126:46057 was elected leader, sending a full tablet report...
I20260810 23:27:39.994625 1241 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.67:44359
I20260810 23:27:39.994817 1459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.67:44359 every 8 connection(s)
I20260810 23:27:39.996227 1241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
I20260810 23:27:40.000983 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 1241
I20260810 23:27:40.001216 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance
I20260810 23:27:40.001863 15037 external_mini_cluster.cc:1428] Setting key b34c3c39a6ba321fdaad48d3260bb6ee
Aug 10 23:27:40 dist-test-slave-rmhg krb5kdc[32308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404459, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:27:40.027879 1468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:27:40.000284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:43235 (local address 127.14.175.126:46057)
0810 23:27:40.001414 (+ 1130us) server_negotiation.cc:206] Beginning negotiation
0810 23:27:40.001421 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:27:40.003515 (+ 2094us) server_negotiation.cc:407] Connection header received
0810 23:27:40.006478 (+ 2963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:27:40.006482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:27:40.006553 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:27:40.006649 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:27:40.008961 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:27:40.010255 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:27:40.012715 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:27:40.013525 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:27:40.013630 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:27:40.018996 (+ 5366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:27:40.019002 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:27:40.019006 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:27:40.019051 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:27:40.023440 (+ 4389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:27:40.024478 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:27:40.024484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:27:40.024488 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:27:40.024705 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:27:40.025430 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:27:40.025436 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:27:40.025440 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:27:40.025864 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:27:40.026048 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0810 23:27:40.026940 (+ 892us) server_negotiation.cc:299] Negotiation successful
0810 23:27:40.027338 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":992,"thread_start_us":628,"threads_started":1}
I20260810 23:27:40.028745 1469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:27:40.001395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46057 (local address 127.14.175.67:43235)
0810 23:27:40.003293 (+ 1898us) negotiation.cc:107] Waiting for socket to connect
0810 23:27:40.003373 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0810 23:27:40.006208 (+ 2835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:27:40.006901 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:27:40.006911 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:27:40.007750 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:27:40.008775 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:27:40.008799 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:27:40.010415 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:27:40.010420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:27:40.012509 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:27:40.012527 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:27:40.013915 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:27:40.013923 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:27:40.014297 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:27:40.015131 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:27:40.015198 (+ 67us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:27:40.018667 (+ 3469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:27:40.023635 (+ 4968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:27:40.023645 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:27:40.023663 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:27:40.024324 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:27:40.024843 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:27:40.024847 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:27:40.024850 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:27:40.025311 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:27:40.026017 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:27:40.026026 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:27:40.026421 (+ 395us) client_negotiation.cc:769] Sending connection context
0810 23:27:40.026688 (+ 267us) client_negotiation.cc:240] Negotiation successful
0810 23:27:40.027274 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1534,"thread_start_us":1055,"threads_started":1}
I20260810 23:27:40.030652 1462 heartbeater.cc:347] Connected to a master server at 127.14.175.126:46057
I20260810 23:27:40.031020 1462 heartbeater.cc:464] Registering TS with master...
I20260810 23:27:40.032032 1462 heartbeater.cc:511] Master 127.14.175.126:46057 requested a full tablet report, sending...
I20260810 23:27:40.033704 623 ts_manager.cc:194] Registered new tserver with Master: 8722e66e949142b182767752dbbcfc1f (127.14.175.67:44359)
I20260810 23:27:40.034756 623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.67@KRBTEST.COM'} at 127.14.175.67:43235
I20260810 23:27:40.043987 15037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:27:40.071081 1468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:27:40.051080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56388 (local address 127.14.175.126:46057)
0810 23:27:40.051241 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 23:27:40.051246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:27:40.052246 (+ 1000us) server_negotiation.cc:407] Connection header received
0810 23:27:40.052460 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:27:40.052465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:27:40.052608 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:27:40.052708 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:27:40.055179 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:27:40.056255 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:27:40.058537 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:27:40.059365 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:27:40.059493 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:27:40.063265 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:27:40.063270 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:27:40.063273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:27:40.063315 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:27:40.067402 (+ 4087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:27:40.068414 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:27:40.068418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:27:40.068421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:27:40.068666 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:27:40.069081 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:27:40.069085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:27:40.069087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:27:40.069561 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:27:40.069975 (+ 414us) server_negotiation.cc:1035] Waiting for connection context
0810 23:27:40.070282 (+ 307us) server_negotiation.cc:299] Negotiation successful
0810 23:27:40.070638 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 23:27:40.076596 623 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:27:40.089699 623 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:27:40.143625 1487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:27:40.109192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:44359 (local address 127.0.0.1:34474)
0810 23:27:40.124399 (+ 15207us) negotiation.cc:107] Waiting for socket to connect
0810 23:27:40.124539 (+ 140us) client_negotiation.cc:174] Beginning negotiation
0810 23:27:40.124893 (+ 354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:27:40.128238 (+ 3345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:27:40.128246 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:27:40.128280 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:27:40.130769 (+ 2489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:27:40.130793 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:27:40.134210 (+ 3417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:27:40.134216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:27:40.138256 (+ 4040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:27:40.138275 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:27:40.141892 (+ 3617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:27:40.141901 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:27:40.142280 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:27:40.142359 (+ 79us) client_negotiation.cc:769] Sending connection context
0810 23:27:40.142506 (+ 147us) client_negotiation.cc:240] Negotiation successful
0810 23:27:40.142791 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14761,"mutex_wait_us":28,"thread_start_us":17515,"threads_started":1}
I20260810 23:27:40.145036 1489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:27:40.110020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34474 (local address 127.14.175.67:44359)
0810 23:27:40.126489 (+ 16469us) server_negotiation.cc:206] Beginning negotiation
0810 23:27:40.126501 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:27:40.126572 (+ 71us) server_negotiation.cc:407] Connection header received
0810 23:27:40.126747 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:27:40.126758 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:27:40.127083 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:27:40.127406 (+ 323us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:27:40.131364 (+ 3958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:27:40.133850 (+ 2486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:27:40.138497 (+ 4647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:27:40.141066 (+ 2569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:27:40.141301 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:27:40.141611 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0810 23:27:40.144016 (+ 2405us) server_negotiation.cc:299] Negotiation successful
0810 23:27:40.144269 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16175,"thread_start_us":15771,"threads_started":1}
I20260810 23:27:40.146199 1486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:27:40.109479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:35165 (local address 127.0.0.1:49178)
0810 23:27:40.127204 (+ 17725us) negotiation.cc:107] Waiting for socket to connect
0810 23:27:40.127226 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:27:40.127544 (+ 318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:27:40.128823 (+ 1279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:27:40.128832 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:27:40.128854 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:27:40.134107 (+ 5253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:27:40.134125 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:27:40.137373 (+ 3248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:27:40.137379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:27:40.140846 (+ 3467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:27:40.140867 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:27:40.144616 (+ 3749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:27:40.144622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:27:40.144838 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:27:40.144864 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 23:27:40.144956 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 23:27:40.145173 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17516,"mutex_wait_us":19,"thread_start_us":14097,"threads_started":1}
I20260810 23:27:40.146799 1488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:27:40.110106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49178 (local address 127.14.175.65:35165)
0810 23:27:40.122724 (+ 12618us) server_negotiation.cc:206] Beginning negotiation
0810 23:27:40.122737 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:27:40.127361 (+ 4624us) server_negotiation.cc:407] Connection header received
0810 23:27:40.127835 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:27:40.127847 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:27:40.128122 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:27:40.128344 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:27:40.134474 (+ 6130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:27:40.137199 (+ 2725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:27:40.141090 (+ 3891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:27:40.143699 (+ 2609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:27:40.143825 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:27:40.144215 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0810 23:27:40.145096 (+ 881us) server_negotiation.cc:299] Negotiation successful
0810 23:27:40.145755 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12352,"thread_start_us":12018,"threads_started":1}
I20260810 23:27:40.149842 1491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:27:40.127350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:43781 (local address 127.0.0.1:60424)
0810 23:27:40.131910 (+ 4560us) negotiation.cc:107] Waiting for socket to connect
0810 23:27:40.131931 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:27:40.132103 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:27:40.132739 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:27:40.132745 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:27:40.132771 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:27:40.135321 (+ 2550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:27:40.135337 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:27:40.138462 (+ 3125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:27:40.138468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:27:40.145414 (+ 6946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:27:40.145432 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:27:40.148287 (+ 2855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:27:40.148298 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:27:40.148569 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:27:40.148606 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 23:27:40.148705 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 23:27:40.148865 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4395,"mutex_wait_us":65,"thread_start_us":4568,"threads_started":1}
I20260810 23:27:40.151396 1492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:27:40.128161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60424 (local address 127.14.175.66:43781)
0810 23:27:40.130230 (+ 2069us) server_negotiation.cc:206] Beginning negotiation
0810 23:27:40.130237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:27:40.132165 (+ 1928us) server_negotiation.cc:407] Connection header received
0810 23:27:40.132265 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:27:40.132271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:27:40.132481 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:27:40.132715 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:27:40.135661 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:27:40.138144 (+ 2483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:27:40.145719 (+ 7575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:27:40.147842 (+ 2123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:27:40.148017 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:27:40.148415 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0810 23:27:40.150256 (+ 1841us) server_negotiation.cc:299] Negotiation successful
0810 23:27:40.150576 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1865,"thread_start_us":1587,"threads_started":1}
I20260810 23:27:40.152578 879 tablet_service.cc:1467] Processing CreateTablet for tablet e7a3f06310a1431ab56234f8721582d3 (DEFAULT_TABLE table=test-table [id=8b306a4735154c8fbfc3d6d776d61560]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:27:40.151289 1349 tablet_service.cc:1467] Processing CreateTablet for tablet e7a3f06310a1431ab56234f8721582d3 (DEFAULT_TABLE table=test-table [id=8b306a4735154c8fbfc3d6d776d61560]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:27:40.154631 879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a3f06310a1431ab56234f8721582d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:27:40.154745 1349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a3f06310a1431ab56234f8721582d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:27:40.156699 1116 tablet_service.cc:1467] Processing CreateTablet for tablet e7a3f06310a1431ab56234f8721582d3 (DEFAULT_TABLE table=test-table [id=8b306a4735154c8fbfc3d6d776d61560]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:27:40.160151 1116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a3f06310a1431ab56234f8721582d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:27:40.175256 1494 tablet_bootstrap.cc:492] T e7a3f06310a1431ab56234f8721582d3 P 3a5269f2083a4b56a9b64a242e309006: Bootstrap starting.
I20260810 23:27:40.178062 1495 tablet_bootstrap.cc:492] T e7a3f06310a1431ab56234f8721582d3 P 8722e66e949142b182767752dbbcfc1f: Bootstrap starting.
I20260810 23:27:40.184478 1496 tablet_bootstrap.cc:492] T e7a3f06310a1431ab56234f8721582d3 P e99115f3ba134d469d53977cceaddb48: Bootstrap starting.
I20260810 23:27:40.184760 1495 tablet_bootstrap.cc:654] T e7a3f06310a1431ab56234f8721582d3 P 8722e66e949142b182767752dbbcfc1f: Neither blocks nor log segments found. Creating new log.
I20260810 23:27:40.184914 1494 tablet_bootstrap.cc:654] T e7a3f06310a1431ab56234f8721582d3 P 3a5269f2083a4b56a9b64a242e309006: Neither blocks nor log segments found. Creating new log.
I20260810 23:27:40.187527 1495 log.cc:824] T e7a3f06310a1431ab56234f8721582d3 P 8722e66e949142b182767752dbbcfc1f: Log is configured to *not* fsync() on all Append() calls
I20260810 23:27:40.187582 1494 log.cc:824] T e7a3f06310a1431ab56234f8721582d3 P 3a5269f2083a4b56a9b64a242e309006: Log is configured to *not* fsync() on all Append() calls
I20260810 23:27:40.189857 1496 tablet_bootstrap.cc:654] T e7a3f06310a1431ab56234f8721582d3 P e99115f3ba134d469d53977cceaddb48: Neither blocks nor log segments found. Creating new log.
I20260810 23:27:40.192377 1496 log.cc:824] T e7a3f06310a1431ab56234f8721582d3 P e99115f3ba134d469d53977cceaddb48: Log is configured to *not* fsync() on all Append() calls
I20260810 23:27:40.193979 1495 tablet_bootstrap.cc:492] T e7a3f06310a1431ab56234f8721582d3 P 8722e66e949142b182767752dbbcfc1f: No bootstrap required, opened a new log
I20260810 23:27:40.195201 1495 ts_tablet_manager.cc:1397] T e7a3f06310a1431ab56234f8721582d3 P 8722e66e949142b182767752dbbcfc1f: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.012s
I20260810 23:27:40.195828 1494 tablet_bootstrap.cc:492] T e7a3f06310a1431ab56234f8721582d3 P 3a5269f2083a4b56a9b64a242e309006: No bootstrap required, opened a new log
I20260810 23:27:40.196274 1494 ts_tablet_manager.cc:1397] T e7a3f06310a1431ab56234f8721582d3 P 3a5269f2083a4b56a9b64a242e309006: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.012s
I20260810 23:27:40.197299 1496 tablet_bootstrap.cc:492] T e7a3f06310a1431ab56234f8721582d3 P e99115f3ba134d469d53977cceaddb48: No bootstrap required, opened a new log
I20260810 23:27:40.197734 1496 ts_tablet_manager.cc:1397] T e7a3f06310a1431ab56234f8721582d3 P e99115f3ba134d469d53977cceaddb48: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260810 23:27:40.209681 1494 raft_consensus.cc:348] T e7a3f06310a1431ab56234f8721582d3 P 3a5269f2083a4b56a9b64a242e309006 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8722e66e949142b182767752dbbcfc1f" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44359 } } peers { permanent_uuid: "3a5269f2083a4b56a9b64a242e309006" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35165 } } peers { permanent_uuid: "e99115f3ba134d469d53977cceaddb48" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 43781 } }
I20260810 23:27:40.210328 1494 raft_consensus.cc:374] T e7a3f06310a1431ab56234f8721582d3 P 3a5269f2083a4b56a9b64a242e309006 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:27:40.210516 1494 raft_consensus.cc:729] T e7a3f06310a1431ab56234f8721582d3 P 3a5269f2083a4b56a9b64a242e309006 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5269f2083a4b56a9b64a242e309006, State: Initialized, Role: FOLLOWER
I20260810 23:27:40.211356 1494 consensus_queue.cc:260] T e7a3f06310a1431ab56234f8721582d3 P 3a5269f2083a4b56a9b64a242e309006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8722e66e949142b182767752dbbcfc1f" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44359 } } peers { permanent_uuid: "3a5269f2083a4b56a9b64a242e309006" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35165 } } peers { permanent_uuid: "e99115f3ba134d469d53977cceaddb48" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 43781 } }
I20260810 23:27:40.212272 1495 raft_consensus.cc:348] T e7a3f06310a1431ab56234f8721582d3 P 8722e66e949142b182767752dbbcfc1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8722e66e949142b182767752dbbcfc1f" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44359 } } peers { permanent_uuid: "3a5269f2083a4b56a9b64a242e309006" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35165 } } peers { permanent_uuid: "e99115f3ba134d469d53977cceaddb48" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 43781 } }
I20260810 23:27:40.213033 1495 raft_consensus.cc:374] T e7a3f06310a1431ab56234f8721582d3 P 8722e66e949142b182767752dbbcfc1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:27:40.213302 1495 raft_consensus.cc:729] T e7a3f06310a1431ab56234f8721582d3 P 8722e66e949142b182767752dbbcfc1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8722e66e949142b182767752dbbcfc1f, State: Initialized, Role: FOLLOWER
I20260810 23:27:40.214238 1495 consensus_queue.cc:260] T e7a3f06310a1431ab56234f8721582d3 P 8722e66e949142b182767752dbbcfc1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8722e66e949142b182767752dbbcfc1f" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44359 } } peers { permanent_uuid: "3a5269f2083a4b56a9b64a242e309006" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35165 } } peers { permanent_uuid: "e99115f3ba134d469d53977cceaddb48" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 43781 } }
I20260810 23:27:40.214708 1494 ts_tablet_manager.cc:1428] T e7a3f06310a1431ab56234f8721582d3 P 3a5269f2083a4b56a9b64a242e309006: Time spent starting tablet: real 0.018s user 0.000s sys 0.019s
I20260810 23:27:40.215004 1496 raft_consensus.cc:348] T e7a3f06310a1431ab56234f8721582d3 P e99115f3ba134d469d53977cceaddb48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8722e66e949142b182767752dbbcfc1f" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44359 } } peers { permanent_uuid: "3a5269f2083a4b56a9b64a242e309006" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35165 } } peers { permanent_uuid: "e99115f3ba134d469d53977cceaddb48" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 43781 } }
I20260810 23:27:40.215767 1496 raft_consensus.cc:374] T e7a3f06310a1431ab56234f8721582d3 P e99115f3ba134d469d53977cceaddb48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:27:40.215950 1496 raft_consensus.cc:729] T e7a3f06310a1431ab56234f8721582d3 P e99115f3ba134d469d53977cceaddb48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e99115f3ba134d469d53977cceaddb48, State: Initialized, Role: FOLLOWER
I20260810 23:27:40.216758 1462 heartbeater.cc:503] Master 127.14.175.126:46057 was elected leader, sending a full tablet report...
I20260810 23:27:40.216935 1496 consensus_queue.cc:260] T e7a3f06310a1431ab56234f8721582d3 P e99115f3ba134d469d53977cceaddb48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8722e66e949142b182767752dbbcfc1f" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44359 } } peers { permanent_uuid: "3a5269f2083a4b56a9b64a242e309006" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35165 } } peers { permanent_uuid: "e99115f3ba134d469d53977cceaddb48" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 43781 } }
I20260810 23:27:40.218152 1495 ts_tablet_manager.cc:1428] T e7a3f06310a1431ab56234f8721582d3 P 8722e66e949142b182767752dbbcfc1f: Time spent starting tablet: real 0.023s user 0.010s sys 0.011s
I20260810 23:27:40.221837 1496 ts_tablet_manager.cc:1428] T e7a3f06310a1431ab56234f8721582d3 P e99115f3ba134d469d53977cceaddb48: Time spent starting tablet: real 0.024s user 0.019s sys 0.004s
W20260810 23:27:40.249562 1465 tablet.cc:2369] T e7a3f06310a1431ab56234f8721582d3 P 8722e66e949142b182767752dbbcfc1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:27:40.260744 987 tablet.cc:2369] T e7a3f06310a1431ab56234f8721582d3 P 3a5269f2083a4b56a9b64a242e309006: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:27:40.305858 1502 raft_consensus.cc:482] T e7a3f06310a1431ab56234f8721582d3 P 8722e66e949142b182767752dbbcfc1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:27:40.306214 1502 raft_consensus.cc:504] T e7a3f06310a1431ab56234f8721582d3 P 8722e66e949142b182767752dbbcfc1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8722e66e949142b182767752dbbcfc1f" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44359 } } peers { permanent_uuid: "3a5269f2083a4b56a9b64a242e309006" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35165 } } peers { permanent_uuid: "e99115f3ba134d469d53977cceaddb48" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 43781 } }
I20260810 23:27:40.308487 1502 leader_election.cc:288] T e7a3f06310a1431ab56234f8721582d3 P 8722e66e949142b182767752dbbcfc1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a5269f2083a4b56a9b64a242e309006 (127.14.175.65:35165), e99115f3ba134d469d53977cceaddb48 (127.14.175.66:43781)
I20260810 23:27:40.317691 1488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:27:40.309293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:53281 (local address 127.14.175.65:35165)
0810 23:27:40.309561 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0810 23:27:40.309566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:27:40.309585 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:27:40.309653 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:27:40.309658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:27:40.309720 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:27:40.309815 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:27:40.310521 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:27:40.311858 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:27:40.315155 (+ 3297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:27:40.316482 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:27:40.316688 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:27:40.316777 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 23:27:40.317188 (+ 411us) server_negotiation.cc:299] Negotiation successful
0810 23:27:40.317328 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260810 23:27:40.317682 1469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:27:40.308814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:35165 (local address 127.14.175.67:53281)
0810 23:27:40.309056 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0810 23:27:40.309085 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 23:27:40.309262 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:27:40.309860 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:27:40.309864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:27:40.309893 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:27:40.310350 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:27:40.310362 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:27:40.312045 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:27:40.312049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:27:40.314928 (+ 2879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:27:40.314943 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:27:40.316664 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:27:40.316671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:27:40.316978 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:27:40.317009 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 23:27:40.317109 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 23:27:40.317285 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":1}
I20260810 23:27:40.319548 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a3f06310a1431ab56234f8721582d3" candidate_uuid: "8722e66e949142b182767752dbbcfc1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5269f2083a4b56a9b64a242e309006" is_pre_election: true
I20260810 23:27:40.320670 910 raft_consensus.cc:2454] T e7a3f06310a1431ab56234f8721582d3 P 3a5269f2083a4b56a9b64a242e309006 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8722e66e949142b182767752dbbcfc1f in term 0.
I20260810 23:27:40.321913 1492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:27:40.309405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:45623 (local address 127.14.175.66:43781)
0810 23:27:40.309729 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0810 23:27:40.309734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:27:40.311562 (+ 1828us) server_negotiation.cc:407] Connection header received
0810 23:27:40.311834 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:27:40.311838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:27:40.311901 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:27:40.311986 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:27:40.314216 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:27:40.315481 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:27:40.318756 (+ 3275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:27:40.320500 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:27:40.320753 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:27:40.320837 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 23:27:40.321389 (+ 552us) server_negotiation.cc:299] Negotiation successful
0810 23:27:40.321552 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:27:40.321863 1278 leader_election.cc:302] T e7a3f06310a1431ab56234f8721582d3 P 8722e66e949142b182767752dbbcfc1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a5269f2083a4b56a9b64a242e309006, 8722e66e949142b182767752dbbcfc1f; no voters:
I20260810 23:27:40.322511 1502 raft_consensus.cc:2781] T e7a3f06310a1431ab56234f8721582d3 P 8722e66e949142b182767752dbbcfc1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:27:40.322875 1502 raft_consensus.cc:482] T e7a3f06310a1431ab56234f8721582d3 P 8722e66e949142b182767752dbbcfc1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:27:40.322944 1507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:27:40.309596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:43781 (local address 127.14.175.67:45623)
0810 23:27:40.311384 (+ 1788us) negotiation.cc:107] Waiting for socket to connect
0810 23:27:40.311429 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 23:27:40.311592 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:27:40.312018 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:27:40.312024 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:27:40.312051 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:27:40.313882 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:27:40.313899 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:27:40.315718 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:27:40.315722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:27:40.318532 (+ 2810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:27:40.318551 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:27:40.320872 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:27:40.320881 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:27:40.321139 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:27:40.321215 (+ 76us) client_negotiation.cc:769] Sending connection context
0810 23:27:40.322133 (+ 918us) client_negotiation.cc:240] Negotiation successful
0810 23:27:40.322292 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1616,"thread_start_us":1331,"threads_started":1}
I20260810 23:27:40.323060 1502 raft_consensus.cc:3037] T e7a3f06310a1431ab56234f8721582d3 P 8722e66e949142b182767752dbbcfc1f [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:27:40.324678 1148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a3f06310a1431ab56234f8721582d3" candidate_uuid: "8722e66e949142b182767752dbbcfc1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e99115f3ba134d469d53977cceaddb48" is_pre_election: true
I20260810 23:27:40.325780 1148 raft_consensus.cc:2454] T e7a3f06310a1431ab56234f8721582d3 P e99115f3ba134d469d53977cceaddb48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8722e66e949142b182767752dbbcfc1f in term 0.
I20260810 23:27:40.327373 1502 raft_consensus.cc:504] T e7a3f06310a1431ab56234f8721582d3 P 8722e66e949142b182767752dbbcfc1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8722e66e949142b182767752dbbcfc1f" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44359 } } peers { permanent_uuid: "3a5269f2083a4b56a9b64a242e309006" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35165 } } peers { permanent_uuid: "e99115f3ba134d469d53977cceaddb48" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 43781 } }
I20260810 23:27:40.328238 1502 leader_election.cc:288] T e7a3f06310a1431ab56234f8721582d3 P 8722e66e949142b182767752dbbcfc1f [CANDIDATE]: Term 1 election: Requested vote from peers 3a5269f2083a4b56a9b64a242e309006 (127.14.175.65:35165), e99115f3ba134d469d53977cceaddb48 (127.14.175.66:43781)
I20260810 23:27:40.328884 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a3f06310a1431ab56234f8721582d3" candidate_uuid: "8722e66e949142b182767752dbbcfc1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5269f2083a4b56a9b64a242e309006"
I20260810 23:27:40.328887 1148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a3f06310a1431ab56234f8721582d3" candidate_uuid: "8722e66e949142b182767752dbbcfc1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e99115f3ba134d469d53977cceaddb48"
I20260810 23:27:40.329200 1148 raft_consensus.cc:3037] T e7a3f06310a1431ab56234f8721582d3 P e99115f3ba134d469d53977cceaddb48 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:27:40.329208 910 raft_consensus.cc:3037] T e7a3f06310a1431ab56234f8721582d3 P 3a5269f2083a4b56a9b64a242e309006 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:27:40.333266 1148 raft_consensus.cc:2454] T e7a3f06310a1431ab56234f8721582d3 P e99115f3ba134d469d53977cceaddb48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8722e66e949142b182767752dbbcfc1f in term 1.
I20260810 23:27:40.333465 910 raft_consensus.cc:2454] T e7a3f06310a1431ab56234f8721582d3 P 3a5269f2083a4b56a9b64a242e309006 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8722e66e949142b182767752dbbcfc1f in term 1.
I20260810 23:27:40.334004 1276 leader_election.cc:302] T e7a3f06310a1431ab56234f8721582d3 P 8722e66e949142b182767752dbbcfc1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8722e66e949142b182767752dbbcfc1f, e99115f3ba134d469d53977cceaddb48; no voters:
I20260810 23:27:40.334509 1502 raft_consensus.cc:2781] T e7a3f06310a1431ab56234f8721582d3 P 8722e66e949142b182767752dbbcfc1f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:27:40.335561 1502 raft_consensus.cc:686] T e7a3f06310a1431ab56234f8721582d3 P 8722e66e949142b182767752dbbcfc1f [term 1 LEADER]: Becoming Leader. State: Replica: 8722e66e949142b182767752dbbcfc1f, State: Running, Role: LEADER
I20260810 23:27:40.336304 1502 consensus_queue.cc:237] T e7a3f06310a1431ab56234f8721582d3 P 8722e66e949142b182767752dbbcfc1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8722e66e949142b182767752dbbcfc1f" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44359 } } peers { permanent_uuid: "3a5269f2083a4b56a9b64a242e309006" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35165 } } peers { permanent_uuid: "e99115f3ba134d469d53977cceaddb48" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 43781 } }
I20260810 23:27:40.343341 621 catalog_manager.cc:5495] T e7a3f06310a1431ab56234f8721582d3 P 8722e66e949142b182767752dbbcfc1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8722e66e949142b182767752dbbcfc1f (127.14.175.67). New cstate: current_term: 1 leader_uuid: "8722e66e949142b182767752dbbcfc1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8722e66e949142b182767752dbbcfc1f" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a5269f2083a4b56a9b64a242e309006" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e99115f3ba134d469d53977cceaddb48" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 43781 } health_report { overall_health: UNKNOWN } } }
I20260810 23:27:40.385630 1489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:27:40.376597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34480 (local address 127.14.175.67:44359)
0810 23:27:40.376805 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 23:27:40.376811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:27:40.377065 (+ 254us) server_negotiation.cc:407] Connection header received
0810 23:27:40.377359 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:27:40.377364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:27:40.377500 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:27:40.377658 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:27:40.378274 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:27:40.379506 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:27:40.382492 (+ 2986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:27:40.383474 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:27:40.383597 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:27:40.384351 (+ 754us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:27:40.384984 (+ 633us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:27:40.385140 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 23:27:40.385255 (+ 115us) server_negotiation.cc:299] Negotiation successful
0810 23:27:40.385333 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 23:27:40.397341 910 raft_consensus.cc:1261] T e7a3f06310a1431ab56234f8721582d3 P 3a5269f2083a4b56a9b64a242e309006 [term 1 FOLLOWER]: Refusing update from remote peer 8722e66e949142b182767752dbbcfc1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:27:40.397460 1148 raft_consensus.cc:1261] T e7a3f06310a1431ab56234f8721582d3 P e99115f3ba134d469d53977cceaddb48 [term 1 FOLLOWER]: Refusing update from remote peer 8722e66e949142b182767752dbbcfc1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:27:40.398789 1508 consensus_queue.cc:1035] T e7a3f06310a1431ab56234f8721582d3 P 8722e66e949142b182767752dbbcfc1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a5269f2083a4b56a9b64a242e309006" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:27:40.399403 1502 consensus_queue.cc:1035] T e7a3f06310a1431ab56234f8721582d3 P 8722e66e949142b182767752dbbcfc1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e99115f3ba134d469d53977cceaddb48" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 43781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 23:27:40.410954 1222 tablet.cc:2369] T e7a3f06310a1431ab56234f8721582d3 P e99115f3ba134d469d53977cceaddb48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:27:40.419768 1517 mvcc.cc:205] Tried to move back new op lower bound from 7317112669773705216 to 7317112669543288832. Current Snapshot: MvccSnapshot[applied={T|T < 7317112669773705216}]
I20260810 23:27:40.442986 621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:27:40.443427 621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:27:40.448977 621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05b36ba2de9e4fe2a34a608f5eb42b9c: Sending DeleteTablet for 3 replicas of tablet e7a3f06310a1431ab56234f8721582d3
I20260810 23:27:40.450695 879 tablet_service.cc:1514] Processing DeleteTablet for tablet e7a3f06310a1431ab56234f8721582d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:27:40 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:49178
I20260810 23:27:40.450834 1349 tablet_service.cc:1514] Processing DeleteTablet for tablet e7a3f06310a1431ab56234f8721582d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:27:40 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:34474
I20260810 23:27:40.450970 1116 tablet_service.cc:1514] Processing DeleteTablet for tablet e7a3f06310a1431ab56234f8721582d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:27:40 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:60424
I20260810 23:27:40.454160 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 765
I20260810 23:27:40.464540 1528 tablet_replica.cc:330] T e7a3f06310a1431ab56234f8721582d3 P 3a5269f2083a4b56a9b64a242e309006: stopping tablet replica
I20260810 23:27:40.465463 1528 raft_consensus.cc:2229] T e7a3f06310a1431ab56234f8721582d3 P 3a5269f2083a4b56a9b64a242e309006 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:27:40.466272 1528 raft_consensus.cc:2258] T e7a3f06310a1431ab56234f8721582d3 P 3a5269f2083a4b56a9b64a242e309006 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:27:40.468904 1528 ts_tablet_manager.cc:1905] T e7a3f06310a1431ab56234f8721582d3 P 3a5269f2083a4b56a9b64a242e309006: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:27:40.479036 1528 ts_tablet_manager.cc:1918] T e7a3f06310a1431ab56234f8721582d3 P 3a5269f2083a4b56a9b64a242e309006: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:27:40.479408 1528 log.cc:1196] T e7a3f06310a1431ab56234f8721582d3 P 3a5269f2083a4b56a9b64a242e309006: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-0/wal/wals/e7a3f06310a1431ab56234f8721582d3
I20260810 23:27:40.480301 1528 ts_tablet_manager.cc:1939] T e7a3f06310a1431ab56234f8721582d3 P 3a5269f2083a4b56a9b64a242e309006: Deleting consensus metadata
I20260810 23:27:40.483747 601 catalog_manager.cc:4841] TS 3a5269f2083a4b56a9b64a242e309006 (127.14.175.65:35165): tablet e7a3f06310a1431ab56234f8721582d3 (table test-table [id=8b306a4735154c8fbfc3d6d776d61560]) successfully deleted
I20260810 23:27:40.486411 1529 tablet_replica.cc:330] T e7a3f06310a1431ab56234f8721582d3 P 8722e66e949142b182767752dbbcfc1f: stopping tablet replica
I20260810 23:27:40.486967 1529 raft_consensus.cc:2229] T e7a3f06310a1431ab56234f8721582d3 P 8722e66e949142b182767752dbbcfc1f [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:27:40.487905 1531 tablet_replica.cc:330] T e7a3f06310a1431ab56234f8721582d3 P e99115f3ba134d469d53977cceaddb48: stopping tablet replica
I20260810 23:27:40.487862 1529 raft_consensus.cc:2258] T e7a3f06310a1431ab56234f8721582d3 P 8722e66e949142b182767752dbbcfc1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:27:40.488531 1531 raft_consensus.cc:2229] T e7a3f06310a1431ab56234f8721582d3 P e99115f3ba134d469d53977cceaddb48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:27:40.489346 1531 raft_consensus.cc:2258] T e7a3f06310a1431ab56234f8721582d3 P e99115f3ba134d469d53977cceaddb48 [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 10 23:27:40 dist-test-slave-rmhg krb5kdc[32308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.65@KRBTEST.COM
I20260810 23:27:40.493639 1529 ts_tablet_manager.cc:1905] T e7a3f06310a1431ab56234f8721582d3 P 8722e66e949142b182767752dbbcfc1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:27:40.495935 1531 ts_tablet_manager.cc:1905] T e7a3f06310a1431ab56234f8721582d3 P e99115f3ba134d469d53977cceaddb48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:27:40.505025 1531 ts_tablet_manager.cc:1918] T e7a3f06310a1431ab56234f8721582d3 P e99115f3ba134d469d53977cceaddb48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:27:40.505483 1531 log.cc:1196] T e7a3f06310a1431ab56234f8721582d3 P e99115f3ba134d469d53977cceaddb48: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-1/wal/wals/e7a3f06310a1431ab56234f8721582d3
I20260810 23:27:40.506433 1531 ts_tablet_manager.cc:1939] T e7a3f06310a1431ab56234f8721582d3 P e99115f3ba134d469d53977cceaddb48: Deleting consensus metadata
I20260810 23:27:40.506704 1529 ts_tablet_manager.cc:1918] T e7a3f06310a1431ab56234f8721582d3 P 8722e66e949142b182767752dbbcfc1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:27:40.506978 1529 log.cc:1196] T e7a3f06310a1431ab56234f8721582d3 P 8722e66e949142b182767752dbbcfc1f: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404299514076-15037-0/minicluster-data/ts-2/wal/wals/e7a3f06310a1431ab56234f8721582d3
I20260810 23:27:40.507071 1488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:27:40.455677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49192 (local address 127.14.175.65:35165)
0810 23:27:40.455890 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 23:27:40.455897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:27:40.479606 (+ 23709us) server_negotiation.cc:407] Connection header received
0810 23:27:40.480240 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:27:40.480248 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:27:40.480353 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:27:40.480436 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:27:40.482439 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:27:40.483242 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:27:40.485085 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:27:40.486111 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:27:40.486384 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:27:40.493903 (+ 7519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:27:40.493932 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:27:40.493952 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:27:40.494201 (+ 249us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:27:40.500349 (+ 6148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:27:40.501771 (+ 1422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:27:40.501785 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:27:40.501789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:27:40.502053 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:27:40.503249 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:27:40.503255 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:27:40.503258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:27:40.503975 (+ 717us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:27:40.504415 (+ 440us) server_negotiation.cc:1035] Waiting for connection context
0810 23:27:40.506095 (+ 1680us) server_negotiation.cc:299] Negotiation successful
0810 23:27:40.506369 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 23:27:40.507798 1529 ts_tablet_manager.cc:1939] T e7a3f06310a1431ab56234f8721582d3 P 8722e66e949142b182767752dbbcfc1f: Deleting consensus metadata
I20260810 23:27:40.508737 975 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:27:40.509282 598 catalog_manager.cc:4841] TS e99115f3ba134d469d53977cceaddb48 (127.14.175.66:43781): tablet e7a3f06310a1431ab56234f8721582d3 (table test-table [id=8b306a4735154c8fbfc3d6d776d61560]) successfully deleted
I20260810 23:27:40.510798 598 catalog_manager.cc:4841] TS 8722e66e949142b182767752dbbcfc1f (127.14.175.67:44359): tablet e7a3f06310a1431ab56234f8721582d3 (table test-table [id=8b306a4735154c8fbfc3d6d776d61560]) successfully deleted
I20260810 23:27:40.610682 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 765
I20260810 23:27:40.640693 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 998
Aug 10 23:27:40 dist-test-slave-rmhg krb5kdc[32308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.66@KRBTEST.COM
I20260810 23:27:40.658635 1492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:27:40.641701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60426 (local address 127.14.175.66:43781)
0810 23:27:40.641917 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 23:27:40.641924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:27:40.641979 (+ 55us) server_negotiation.cc:407] Connection header received
0810 23:27:40.642054 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:27:40.642060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:27:40.642162 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:27:40.642278 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:27:40.643742 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:27:40.645046 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:27:40.646672 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:27:40.647209 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:27:40.647338 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:27:40.652103 (+ 4765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:27:40.652123 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:27:40.652142 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:27:40.652212 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:27:40.655396 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:27:40.656232 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:27:40.656239 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:27:40.656241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:27:40.656376 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:27:40.656757 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:27:40.656760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:27:40.656762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:27:40.657462 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:27:40.657701 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0810 23:27:40.658036 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 23:27:40.658278 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 23:27:40.659456 1213 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:27:40.806234 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 998
I20260810 23:27:40.832273 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 1241
Aug 10 23:27:40 dist-test-slave-rmhg krb5kdc[32308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.67@KRBTEST.COM
I20260810 23:27:40.851679 1489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:27:40.833182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34488 (local address 127.14.175.67:44359)
0810 23:27:40.833353 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 23:27:40.833357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:27:40.833413 (+ 56us) server_negotiation.cc:407] Connection header received
0810 23:27:40.833615 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:27:40.833619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:27:40.833724 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:27:40.833816 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:27:40.835692 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:27:40.836507 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:27:40.838568 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:27:40.839052 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:27:40.839177 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:27:40.845393 (+ 6216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:27:40.845410 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:27:40.845429 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:27:40.845480 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:27:40.848558 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:27:40.849578 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:27:40.849585 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:27:40.849588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:27:40.849757 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:27:40.850196 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:27:40.850199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:27:40.850201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:27:40.850578 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:27:40.850826 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0810 23:27:40.851205 (+ 379us) server_negotiation.cc:299] Negotiation successful
0810 23:27:40.851383 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 23:27:40.852377 1455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:27:40.973583 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 1241
I20260810 23:27:40.999101 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 566
I20260810 23:27:41.000347 676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:27:41.113387 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 566
I20260810 23:27:41.152717 15037 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 23:27:41.488852 15037 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 23:27:41.489171 15037 mini_ranger.cc:66] Stopping Ranger...
I20260810 23:27:41.835469 15037 mini_ranger.cc:68] Stopped Ranger
2026-08-10 23:27:41.835 UTC [32350] LOG: received smart shutdown request
2026-08-10 23:27:41.839 UTC [32350] LOG: background worker "logical replication launcher" (PID 32366) exited with exit code 1
2026-08-10 23:27:41.840 UTC [32356] LOG: shutting down
2026-08-10 23:27:42.009 UTC [32350] LOG: database system is shut down
2026-08-10T23:27:42Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (54294 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:27:42 dist-test-slave-rmhg krb5kdc[1562](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:27:42 dist-test-slave-rmhg krb5kdc[1562](info): set up 2 sockets
Aug 10 23:27:42 dist-test-slave-rmhg krb5kdc[1562](info): commencing operation
krb5kdc: starting...
W20260810 23:27:43.390527 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.221s user 0.004s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:27:43 dist-test-slave-rmhg krb5kdc[1562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:27:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:27:43Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.175.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.175.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.175.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/rangeradmin_127.14.175.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.175.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/rangeradmin_127.14.175.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.175.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.175.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.175.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/rangerlookup_127.14.175.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.175.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/rangerlookup_127.14.175.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.175.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.175.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.175.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/HTTP_127.14.175.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.175.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/HTTP_127.14.175.84@KRBTEST.COM.keytab.
I20260810 23:27:43.803650 15037 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskwnDIbQ/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/postgres -l logfile start
2026-08-10 23:27:46.287 UTC [1613] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 23:27:46.287 UTC [1613] LOG: listening on IPv4 address "127.14.175.84", port 44107
2026-08-10 23:27:46.290 UTC [1613] LOG: listening on Unix socket "/tmp/.s.PGSQL.44107"
2026-08-10 23:27:46.313 UTC [1615] LOG: database system was shut down at 2026-08-10 23:27:44 UTC
2026-08-10 23:27:46.317 UTC [1613] LOG: database system is ready to accept connections
I20260810 23:27:47.411720 15037 mini_postgres.cc:96] Postgres bound to 44107
2026-08-10 23:27:47.433 UTC [1633] FATAL: database "slave" does not exist
127.14.175.84:44107 - accepting connections
I20260810 23:27:47.434468 15037 mini_ranger.cc:161] Starting Ranger...
I20260810 23:27:47.465842 15037 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 23:27:47.958101 15037 mini_ranger.cc:87] Created ranger Postgres database
I20260810 23:27:47.958217 15037 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ranger-admin
2026-08-10 23:27:48,258 [I] DB FLAVOR :POSTGRES
2026-08-10 23:27:48,259 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 23:27:48,259 [I] Checking connection..
2026-08-10 23:27:48,259 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:44107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 23:27:48,516 [I] Checking connection passed.
2026-08-10 23:27:48,516 [I] --------- Verifying version history table ---------
2026-08-10 23:27:48,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:44107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:27:48,770 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 23:27:48,770 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:44107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:27:49,038 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 23:27:49,038 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 23:27:49,038 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:44107/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 23:27:49.290 UTC [1744] WARNING: there is no transaction in progress
2026-08-10 23:27:49,300 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 23:27:49,301 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:44107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:27:49,555 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 23:27:49,556 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 23:27:49,556 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:44107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 23:27:49,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:44107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 23:27:50,076 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:44107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rmhg.c.gcp-upstream.internal','N') ;"
2026-08-10 23:27:50,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:44107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 23:27:50,607 [I] Table x_portal_user does not exist in database ranger
2026-08-10 23:27:50,608 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:44107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 23:27:50,857 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 23:27:50,857 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:44107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 23:27:51,112 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:44107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 23:27:51,378 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 23:27:51,378 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:44107/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 23:27:52.389 UTC [1923] WARNING: there is no transaction in progress
2026-08-10 23:27:52.406 UTC [1923] WARNING: there is no transaction in progress
2026-08-10 23:27:52.422 UTC [1923] WARNING: there is no transaction in progress
2026-08-10 23:27:52.438 UTC [1923] WARNING: there is no transaction in progress
2026-08-10 23:27:52.453 UTC [1923] WARNING: there is no transaction in progress
2026-08-10 23:27:52.471 UTC [1923] WARNING: there is no transaction in progress
2026-08-10 23:27:52.536 UTC [1923] WARNING: there is no transaction in progress
2026-08-10 23:27:52.542 UTC [1923] WARNING: there is no transaction in progress
2026-08-10 23:27:52.555 UTC [1923] WARNING: there is no transaction in progress
2026-08-10 23:27:52.566 UTC [1923] WARNING: there is no transaction in progress
2026-08-10 23:27:52.577 UTC [1923] WARNING: there is no transaction in progress
2026-08-10 23:27:52.592 UTC [1923] WARNING: there is no transaction in progress
2026-08-10 23:27:52.602 UTC [1923] WARNING: there is no transaction in progress
2026-08-10 23:27:52.609 UTC [1923] WARNING: there is no transaction in progress
2026-08-10 23:27:52.616 UTC [1923] WARNING: there is no transaction in progress
2026-08-10 23:27:52.623 UTC [1923] WARNING: there is no transaction in progress
2026-08-10 23:27:53,248 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 23:27:53,248 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:44107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 23:27:53,508 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 23:27:53,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:44107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 23:27:53,786 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 23:27:53,786 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:44107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 23:27:54,061 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 23:27:54,061 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:44107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 23:27:54,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:44107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 23:27:54,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:44107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rmhg.c.gcp-upstream.internal';"
2026-08-10 23:27:54,820 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 23:27:54,820 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:44107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 23:27:55,087 [I] DB_PATCHES have already been applied
I20260810 23:27:55.093729 15037 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ranger-admin:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskwnDIbQ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 23:27:55.093825 15037 mini_ranger.cc:193] Using host: 127.14.175.84
I20260810 23:27:55.113457 15037 mini_ranger.cc:227] Ranger admin URL: http://127.14.175.84:35643
Aug 10, 2026 11:27:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 11:27:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 11:27:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 11:27:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 11:27:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 11:27:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 11:27:56 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 23:27:56.365237 15037 mini_ranger.cc:160] Time spent starting Ranger: real 8.931s user 0.002s sys 0.061s
Aug 10, 2026 11:28:17 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 11:28:17 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 11:28:17 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 11:28:17 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 11:28:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 11:28:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 11:28:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 11:28:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 11:28:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 11:28:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 11:28:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 11:28:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 11:28:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 11:28:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 11:28:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 11:28:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 11:28:17 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 11:28:18 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 23:28:18.978012 15037 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.14.175.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.14.175.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.14.175.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/rangerkms_127.14.175.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.14.175.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/rangerkms_127.14.175.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.175.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.14.175.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.175.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/HTTP_127.14.175.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.175.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/HTTP_127.14.175.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 23:28:19 dist-test-slave-rmhg krb5kdc[1562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404499, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 23:28:19.177359 15037 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 23:28:19.204473 15037 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 23:28:20.071463 15037 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 23:28:20.071591 15037 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ranger-kms
I20260810 23:28:20.071631 15037 mini_ranger_kms.cc:227] Using postgres at 127.14.175.84:44107
2026-08-10 23:28:20,420 [I] DB FLAVOR :POSTGRES
2026-08-10 23:28:20,421 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 23:28:20,421 [I] Checking connection
2026-08-10 23:28:20,421 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:44107/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 23:28:20,678 [I] connection success
2026-08-10 23:28:20,679 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 23:28:20,679 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 23:28:20,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:44107/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 23:28:20,936 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 23:28:20,937 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 23:28:20,937 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 23:28:20,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.175.84:44107/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 23:28:21,212 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 23:28:21.695400 15037 mini_ranger_kms.cc:310] Created kms service
I20260810 23:28:21.915676 15037 mini_ranger_kms.cc:326] Created kudu user
I20260810 23:28:21.969430 15037 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 23:28:21 dist-test-slave-rmhg krb5kdc[1562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404499, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.14.175.84@KRBTEST.COM
I20260810 23:28:22.467958 15037 mini_ranger_kms.cc:384] Added ranger policy
I20260810 23:28:22.468359 15037 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ranger-kms:/tmp/dist-test-taskwnDIbQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskwnDIbQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 23:28:22.468422 15037 mini_ranger_kms.cc:240] Using host: 127.14.175.84
I20260810 23:28:22.494949 15037 mini_ranger_kms.cc:276] Ranger KMS PID: 2207
I20260810 23:28:22.495081 15037 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.14.175.84:37605
Aug 10, 2026 11:28:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 11:28:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 11:28:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 11:28:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 11:28:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 11:28:24 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 23:28:24 dist-test-slave-rmhg krb5kdc[1562](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786404504, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.175.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:28:25.126411 15037 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.949s user 0.068s sys 0.318s
I20260810 23:28:25.126729 15037 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 23:28:25.126924 15037 mini_ranger_kms.cc:401] 127.14.175.84:37605/kms/v1/keys
Aug 10 23:28:25 dist-test-slave-rmhg krb5kdc[1562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404504, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.175.84@KRBTEST.COM for HTTP/127.14.175.84@KRBTEST.COM
Aug 10 23:28:25 dist-test-slave-rmhg krb5kdc[1562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404504, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.175.84@KRBTEST.COM for HTTP/127.14.175.84@KRBTEST.COM
Aug 10 23:28:28 dist-test-slave-rmhg krb5kdc[1562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.14.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
I20260810 23:28:28.153684 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:34467
--webserver_interface=127.14.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:38295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:34467
--ranger_config_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.175.84:37605/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:28:28.508705 2288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:28:28.509048 2288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:28:28.509197 2288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:28:28.516589 2288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:28:28.516729 2288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:28:28.516811 2288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:28:28.516893 2288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:28:28.516955 2288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:28:28.527976 2288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:38295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/master-0/wal
--ranger_kms_url=127.14.175.84:37605/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:34467
--ranger_config_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:34467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:28:28.529505 2288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:28:28.531016 2288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:28:28.544392 2288 server_base.cc:1034] running on GCE node
W20260810 23:28:28.544312 2304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:28.544356 2303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:28.544692 2308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:28:28.545660 2288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:28:28.547284 2288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:28:28.548543 2288 hybrid_clock.cc:648] HybridClock initialized: now 1786404508548465 us; error 35 us; skew 500 ppm
I20260810 23:28:28.549008 2288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:28:28 dist-test-slave-rmhg krb5kdc[1562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404508, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:28:28.552392 2288 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:28:28.554052 2288 webserver.cc:460] Webserver started at http://127.14.175.126:38567/ using document root <none> and password file <none>
I20260810 23:28:28.554754 2288 fs_manager.cc:359] Metadata directory not provided
I20260810 23:28:28.554876 2288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:28:28.555191 2288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:28:28 dist-test-slave-rmhg krb5kdc[1562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404508, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for HTTP/127.14.175.84@KRBTEST.COM
I20260810 23:28:28.618201 2288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "ff261d817152432bb0581bfb48106ed3"
format_stamp: "Formatted at 2026-08-10 23:28:28 on dist-test-slave-rmhg"
server_key: "8f7d778a29c893684e787ad1acfbb9d9"
server_key_iv: "99a8b347e5eecc76eb3d9c7edea5428c"
server_key_version: "kuduclusterkey@0"
I20260810 23:28:28.619488 2288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "ff261d817152432bb0581bfb48106ed3"
format_stamp: "Formatted at 2026-08-10 23:28:28 on dist-test-slave-rmhg"
server_key: "8f7d778a29c893684e787ad1acfbb9d9"
server_key_iv: "99a8b347e5eecc76eb3d9c7edea5428c"
server_key_version: "kuduclusterkey@0"
I20260810 23:28:28.627687 2288 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260810 23:28:28.650132 2320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:28.652122 2288 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:28:28.652387 2288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "ff261d817152432bb0581bfb48106ed3"
format_stamp: "Formatted at 2026-08-10 23:28:28 on dist-test-slave-rmhg"
server_key: "8f7d778a29c893684e787ad1acfbb9d9"
server_key_iv: "99a8b347e5eecc76eb3d9c7edea5428c"
server_key_version: "kuduclusterkey@0"
I20260810 23:28:28.652719 2288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:28:28.687955 2288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:28:28.693122 2288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:28:28.693599 2288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:28:28.713557 2397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:34467 every 8 connection(s)
I20260810 23:28:28.713601 2288 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:34467
I20260810 23:28:28.715539 2288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:28:28.716992 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 2288
I20260810 23:28:28.717278 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:28:28.718138 15037 external_mini_cluster.cc:1428] Setting key a5575da003e2b942645250fb86d193f3
I20260810 23:28:28.720920 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:28:28.738335 2398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3: Bootstrap starting.
I20260810 23:28:28.742630 2398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3: Neither blocks nor log segments found. Creating new log.
I20260810 23:28:28.744199 2398 log.cc:824] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:28:28 dist-test-slave-rmhg krb5kdc[1562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:28:28.748229 2398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3: No bootstrap required, opened a new log
I20260810 23:28:28.757449 2402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:28.720827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38260 (local address 127.14.175.126:34467)
0810 23:28:28.729829 (+ 9002us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:28.729854 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:28.729938 (+ 84us) server_negotiation.cc:407] Connection header received
0810 23:28:28.732427 (+ 2489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:28.732464 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:28.733077 (+ 613us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:28.733655 (+ 578us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:28.735426 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:28.737596 (+ 2170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:28.739673 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:28.740824 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:28.741036 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:28.746503 (+ 5467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:28.746517 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:28.746542 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:28.746602 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:28.750901 (+ 4299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:28.751939 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:28.751954 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:28.751965 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:28.752192 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:28.752694 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:28.752700 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:28.752703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:28.753620 (+ 917us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:28.753939 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:28.755321 (+ 1382us) server_negotiation.cc:299] Negotiation successful
0810 23:28:28.755854 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8674,"thread_start_us":8404,"threads_started":1}
I20260810 23:28:28.759191 2398 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff261d817152432bb0581bfb48106ed3" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34467 } }
I20260810 23:28:28.759658 2398 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:28:28.759778 2398 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff261d817152432bb0581bfb48106ed3, State: Initialized, Role: FOLLOWER
I20260810 23:28:28.760432 2398 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff261d817152432bb0581bfb48106ed3" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34467 } }
I20260810 23:28:28.760699 2398 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:28:28.760871 2398 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:28:28.761083 2398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:28:28.763504 2398 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff261d817152432bb0581bfb48106ed3" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34467 } }
I20260810 23:28:28.764073 2398 leader_election.cc:302] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff261d817152432bb0581bfb48106ed3; no voters:
I20260810 23:28:28.764715 2398 leader_election.cc:288] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:28:28.764901 2405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:28:28.768314 2398 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:28:28.773936 2405 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [term 1 LEADER]: Becoming Leader. State: Replica: ff261d817152432bb0581bfb48106ed3, State: Running, Role: LEADER
I20260810 23:28:28.774643 2405 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff261d817152432bb0581bfb48106ed3" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34467 } }
I20260810 23:28:28.777901 2408 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff261d817152432bb0581bfb48106ed3. Latest consensus state: current_term: 1 leader_uuid: "ff261d817152432bb0581bfb48106ed3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff261d817152432bb0581bfb48106ed3" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34467 } } }
I20260810 23:28:28.778308 2408 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [sys.catalog]: This master's current role is: LEADER
I20260810 23:28:28.779044 2407 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff261d817152432bb0581bfb48106ed3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff261d817152432bb0581bfb48106ed3" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34467 } } }
I20260810 23:28:28.779364 2407 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [sys.catalog]: This master's current role is: LEADER
I20260810 23:28:28.779903 2398 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 23:28:28.782229 2428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:28:28.795961 2428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:28:28.824445 2428 catalog_manager.cc:1347] Generated new cluster ID: 73d684310dc84ad2890fa69730791bc2
I20260810 23:28:28.824611 2428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:28:28.842025 2428 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:28:28.844389 2428 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:28:28.852840 2428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3: Generated new TSK 0
I20260810 23:28:28.854292 2428 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
I20260810 23:28:29.645474 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.175.84:37605/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.65:0
--local_ip_for_outbound_sockets=127.14.175.65
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:34467
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:38295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:28:29 dist-test-slave-rmhg krb5kdc[1562](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786404509, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 23:28:30.044032 2477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:28:30.044344 2477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:28:30.044528 2477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:28:30.052045 2477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:28:30.052249 2477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:28:30.052452 2477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.65
I20260810 23:28:30.064131 2477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:38295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.14.175.84:37605/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:34467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.65
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:28:30.066063 2477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:28:30.067911 2477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:28:30.087435 2493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:30.092461 2494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:28:30.093927 2477 server_base.cc:1034] running on GCE node
W20260810 23:28:30.094549 2500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:28:30.095568 2477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:28:30.096920 2477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:28:30.098250 2477 hybrid_clock.cc:648] HybridClock initialized: now 1786404510098166 us; error 66 us; skew 500 ppm
I20260810 23:28:30.098742 2477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:28:30 dist-test-slave-rmhg krb5kdc[1562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404510, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:28:30.103658 2477 init.cc:377] Logged in from keytab as kudu/127.14.175.65@KRBTEST.COM (short username kudu)
I20260810 23:28:30.106762 2477 webserver.cc:460] Webserver started at http://127.14.175.65:33365/ using document root <none> and password file <none>
I20260810 23:28:30.107779 2477 fs_manager.cc:359] Metadata directory not provided
I20260810 23:28:30.107928 2477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:28:30.108299 2477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:28:30 dist-test-slave-rmhg krb5kdc[1562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404510, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for HTTP/127.14.175.84@KRBTEST.COM
I20260810 23:28:30.129029 2477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-0/data/instance:
uuid: "04e6d9f68e874b24b86fa0750afbed0b"
format_stamp: "Formatted at 2026-08-10 23:28:30 on dist-test-slave-rmhg"
server_key: "9ef4cadbd9673b49be9e9a6066533b3f"
server_key_iv: "257b0df88c0128ced7365d8bcee5c93e"
server_key_version: "kuduclusterkey@0"
I20260810 23:28:30.130509 2477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance:
uuid: "04e6d9f68e874b24b86fa0750afbed0b"
format_stamp: "Formatted at 2026-08-10 23:28:30 on dist-test-slave-rmhg"
server_key: "9ef4cadbd9673b49be9e9a6066533b3f"
server_key_iv: "257b0df88c0128ced7365d8bcee5c93e"
server_key_version: "kuduclusterkey@0"
I20260810 23:28:30.158754 2477 fs_manager.cc:691] Time spent creating directory manager: real 0.027s user 0.008s sys 0.016s
I20260810 23:28:30.180249 2510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:30.182561 2477 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260810 23:28:30.182801 2477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-0/wal
uuid: "04e6d9f68e874b24b86fa0750afbed0b"
format_stamp: "Formatted at 2026-08-10 23:28:30 on dist-test-slave-rmhg"
server_key: "9ef4cadbd9673b49be9e9a6066533b3f"
server_key_iv: "257b0df88c0128ced7365d8bcee5c93e"
server_key_version: "kuduclusterkey@0"
I20260810 23:28:30.183112 2477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:28:30.247357 2477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:28:30.253180 2477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:28:30.253558 2477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:28:30.255882 2477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:28:30.257939 2477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:28:30.258121 2477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 23:28:30.258335 2477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:28:30.258458 2477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:28:30 dist-test-slave-rmhg krb5kdc[1562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404509, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for HTTP/127.14.175.84@KRBTEST.COM
I20260810 23:28:30.626590 2477 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.65:36571
I20260810 23:28:30.626636 2700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.65:36571 every 8 connection(s)
I20260810 23:28:30.628470 2477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
I20260810 23:28:30.632963 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 2477
I20260810 23:28:30.633237 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance
I20260810 23:28:30.634097 15037 external_mini_cluster.cc:1428] Setting key b4dee0f1f34d116394b4b04a4c791115
Aug 10 23:28:30 dist-test-slave-rmhg krb5kdc[1562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404510, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:28:30.661715 2708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:30.635002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:36847 (local address 127.14.175.126:34467)
0810 23:28:30.635566 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:30.635572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:30.640661 (+ 5089us) server_negotiation.cc:407] Connection header received
0810 23:28:30.640738 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:30.640745 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:30.640845 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:30.640950 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:30.643282 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:30.644408 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:30.647015 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:30.647680 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:30.647803 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:30.652783 (+ 4980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:30.652793 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:30.652798 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:30.652862 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:30.656766 (+ 3904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:30.658201 (+ 1435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:30.658210 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:30.658214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:30.658419 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:30.659221 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:30.659229 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:30.659232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:30.659633 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:30.659779 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:30.660969 (+ 1190us) server_negotiation.cc:299] Negotiation successful
0810 23:28:30.661297 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":1779,"threads_started":1}
I20260810 23:28:30.664359 2707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:30.632137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:34467 (local address 127.14.175.65:36847)
0810 23:28:30.637762 (+ 5625us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:30.637831 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:30.640238 (+ 2407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:30.641472 (+ 1234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:30.641486 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:30.642163 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:28:30.643042 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:30.643073 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:30.644678 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:30.644683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:30.646806 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:30.646824 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:30.648225 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:30.648231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:30.648561 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:30.649444 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:28:30.649477 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:28:30.652396 (+ 2919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:28:30.657062 (+ 4666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:30.657074 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:30.657098 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:30.657875 (+ 777us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:30.658660 (+ 785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:30.658665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:30.658669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:30.659002 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:30.659890 (+ 888us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:28:30.659899 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:28:30.660433 (+ 534us) client_negotiation.cc:769] Sending connection context
0810 23:28:30.662094 (+ 1661us) client_negotiation.cc:240] Negotiation successful
0810 23:28:30.662777 (+ 683us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5301,"thread_start_us":5067,"threads_started":1}
I20260810 23:28:30.666652 2703 heartbeater.cc:347] Connected to a master server at 127.14.175.126:34467
I20260810 23:28:30.667047 2703 heartbeater.cc:464] Registering TS with master...
I20260810 23:28:30.668136 2703 heartbeater.cc:511] Master 127.14.175.126:34467 requested a full tablet report, sending...
I20260810 23:28:30.671031 2347 ts_manager.cc:194] Registered new tserver with Master: 04e6d9f68e874b24b86fa0750afbed0b (127.14.175.65:36571)
WARNING: no policy specified for kudu/127.14.175.66@KRBTEST.COM; defaulting to no policy
I20260810 23:28:30.673177 2347 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.65@KRBTEST.COM'} at 127.14.175.65:36847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
I20260810 23:28:30.733022 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.175.84:37605/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.66:0
--local_ip_for_outbound_sockets=127.14.175.66
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:34467
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:38295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:28:30 dist-test-slave-rmhg krb5kdc[1562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404509, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for HTTP/127.14.175.84@KRBTEST.COM
W20260810 23:28:31.224881 2715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:28:31.225517 2715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:28:31.225827 2715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:28:31.239245 2715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:28:31.239405 2715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:28:31.239648 2715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.66
I20260810 23:28:31.251027 2715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:38295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.14.175.84:37605/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:34467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.66
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:28:31.252528 2715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:28:31.253986 2715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:28:31.270061 2744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:31.273582 2748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:28:31.279064 2715 server_base.cc:1034] running on GCE node
W20260810 23:28:31.279625 2752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:28:31.280498 2715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:28:31.281723 2715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:28:31.283006 2715 hybrid_clock.cc:648] HybridClock initialized: now 1786404511282946 us; error 61 us; skew 500 ppm
I20260810 23:28:31.283366 2715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:28:31 dist-test-slave-rmhg krb5kdc[1562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404511, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:28:31.286861 2715 init.cc:377] Logged in from keytab as kudu/127.14.175.66@KRBTEST.COM (short username kudu)
I20260810 23:28:31.288822 2715 webserver.cc:460] Webserver started at http://127.14.175.66:36097/ using document root <none> and password file <none>
I20260810 23:28:31.289666 2715 fs_manager.cc:359] Metadata directory not provided
I20260810 23:28:31.289892 2715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:28:31.290234 2715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:28:31 dist-test-slave-rmhg krb5kdc[1562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404511, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for HTTP/127.14.175.84@KRBTEST.COM
I20260810 23:28:31.308260 2715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-1/data/instance:
uuid: "eea228eefebc4347acb2db74799297a4"
format_stamp: "Formatted at 2026-08-10 23:28:31 on dist-test-slave-rmhg"
server_key: "e52eaaca65aac94652a0e42dc9a346a1"
server_key_iv: "fbbb1bbad22074d398ca954e8281e3e0"
server_key_version: "kuduclusterkey@0"
I20260810 23:28:31.309480 2715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance:
uuid: "eea228eefebc4347acb2db74799297a4"
format_stamp: "Formatted at 2026-08-10 23:28:31 on dist-test-slave-rmhg"
server_key: "e52eaaca65aac94652a0e42dc9a346a1"
server_key_iv: "fbbb1bbad22074d398ca954e8281e3e0"
server_key_version: "kuduclusterkey@0"
I20260810 23:28:31.331502 2715 fs_manager.cc:691] Time spent creating directory manager: real 0.022s user 0.012s sys 0.004s
I20260810 23:28:31.356689 2761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:31.358841 2715 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.001s
I20260810 23:28:31.359135 2715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-1/wal
uuid: "eea228eefebc4347acb2db74799297a4"
format_stamp: "Formatted at 2026-08-10 23:28:31 on dist-test-slave-rmhg"
server_key: "e52eaaca65aac94652a0e42dc9a346a1"
server_key_iv: "fbbb1bbad22074d398ca954e8281e3e0"
server_key_version: "kuduclusterkey@0"
I20260810 23:28:31.359454 2715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:28:31.405930 2715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:28:31.411423 2715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:28:31.411775 2715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:28:31.413033 2715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:28:31.414954 2715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:28:31.415060 2715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:31.415239 2715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:28:31.415306 2715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:31.453436 2715 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.66:36905
I20260810 23:28:31.453457 2947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.66:36905 every 8 connection(s)
I20260810 23:28:31.455261 2715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
I20260810 23:28:31.459828 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 2715
I20260810 23:28:31.460021 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance
I20260810 23:28:31.460662 15037 external_mini_cluster.cc:1428] Setting key cf0480e04f80e36c788ace07e3896c8b
Aug 10 23:28:31 dist-test-slave-rmhg krb5kdc[1562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404511, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:28:31.503701 2950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:31.458338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:52161 (local address 127.14.175.126:34467)
0810 23:28:31.473675 (+ 15337us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:31.473683 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:31.483027 (+ 9344us) server_negotiation.cc:407] Connection header received
0810 23:28:31.485769 (+ 2742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:31.485776 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:31.485872 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:31.486037 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:31.488145 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:31.489277 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:31.491694 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:31.492341 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:31.492443 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:31.497255 (+ 4812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:31.497263 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:31.497267 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:31.497312 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:31.499972 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:31.500958 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:31.500964 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:31.500967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:31.501109 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:31.501673 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:31.501679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:31.501682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:31.501979 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:31.502118 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:31.503081 (+ 963us) server_negotiation.cc:299] Negotiation successful
0810 23:28:31.503323 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15157,"thread_start_us":18871,"threads_started":1}
I20260810 23:28:31.505036 2952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:31.459034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:34467 (local address 127.14.175.66:52161)
0810 23:28:31.482784 (+ 23750us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:31.482852 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:31.485362 (+ 2510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:31.486322 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:31.486334 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:31.487029 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:28:31.487910 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:31.487938 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:31.489516 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:31.489522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:31.491493 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:31.491523 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:31.492806 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:31.492812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:31.493128 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:31.493951 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:28:31.493981 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:28:31.496892 (+ 2911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:28:31.500137 (+ 3245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:31.500148 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:31.500170 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:31.500834 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:31.501252 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:31.501257 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:31.501261 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:31.501563 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:31.502130 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:28:31.502139 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:28:31.502642 (+ 503us) client_negotiation.cc:769] Sending connection context
0810 23:28:31.502978 (+ 336us) client_negotiation.cc:240] Negotiation successful
0810 23:28:31.503546 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23368,"thread_start_us":23114,"threads_started":1}
I20260810 23:28:31.507051 2948 heartbeater.cc:347] Connected to a master server at 127.14.175.126:34467
WARNING: no policy specified for kudu/127.14.175.67@KRBTEST.COM; defaulting to no policy
I20260810 23:28:31.507405 2948 heartbeater.cc:464] Registering TS with master...
I20260810 23:28:31.508347 2948 heartbeater.cc:511] Master 127.14.175.126:34467 requested a full tablet report, sending...
I20260810 23:28:31.509931 2347 ts_manager.cc:194] Registered new tserver with Master: eea228eefebc4347acb2db74799297a4 (127.14.175.66:36905)
I20260810 23:28:31.510743 2347 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.66@KRBTEST.COM'} at 127.14.175.66:52161
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
I20260810 23:28:31.544481 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.175.84:37605/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.67:0
--local_ip_for_outbound_sockets=127.14.175.67
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:34467
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:38295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:28:31.677532 2703 heartbeater.cc:503] Master 127.14.175.126:34467 was elected leader, sending a full tablet report...
W20260810 23:28:31.911391 2955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:28:31.911707 2955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:28:31.911813 2955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:28:31.919190 2955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:28:31.919325 2955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:28:31.919446 2955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.67
I20260810 23:28:31.929559 2955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:38295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.14.175.84:37605/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:34467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.67
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:28:31.930936 2955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:28:31.932293 2955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:28:31.945825 2965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:31.945864 2964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:31.945902 2967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:28:31.946962 2955 server_base.cc:1034] running on GCE node
I20260810 23:28:31.947585 2955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:28:31.948632 2955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:28:31.949848 2955 hybrid_clock.cc:648] HybridClock initialized: now 1786404511949794 us; error 47 us; skew 500 ppm
I20260810 23:28:31.950232 2955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:28:31 dist-test-slave-rmhg krb5kdc[1562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404511, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:28:31.953727 2955 init.cc:377] Logged in from keytab as kudu/127.14.175.67@KRBTEST.COM (short username kudu)
I20260810 23:28:31.955319 2955 webserver.cc:460] Webserver started at http://127.14.175.67:41201/ using document root <none> and password file <none>
I20260810 23:28:31.956048 2955 fs_manager.cc:359] Metadata directory not provided
I20260810 23:28:31.956172 2955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:28:31.956480 2955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:28:31 dist-test-slave-rmhg krb5kdc[1562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404511, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for HTTP/127.14.175.84@KRBTEST.COM
I20260810 23:28:31.972065 2955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-2/data/instance:
uuid: "aab0adaa90ca48bdb5b49c981a74e557"
format_stamp: "Formatted at 2026-08-10 23:28:31 on dist-test-slave-rmhg"
server_key: "e50d3c9e9daabbe1d0a624d3b715903d"
server_key_iv: "56bdf0d5a593dd886fa00f8623ffad5f"
server_key_version: "kuduclusterkey@0"
I20260810 23:28:31.973069 2955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance:
uuid: "aab0adaa90ca48bdb5b49c981a74e557"
format_stamp: "Formatted at 2026-08-10 23:28:31 on dist-test-slave-rmhg"
server_key: "e50d3c9e9daabbe1d0a624d3b715903d"
server_key_iv: "56bdf0d5a593dd886fa00f8623ffad5f"
server_key_version: "kuduclusterkey@0"
I20260810 23:28:31.978785 2955 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 23:28:31.994159 2979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:31.995817 2955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:28:31.995991 2955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-2/wal
uuid: "aab0adaa90ca48bdb5b49c981a74e557"
format_stamp: "Formatted at 2026-08-10 23:28:31 on dist-test-slave-rmhg"
server_key: "e50d3c9e9daabbe1d0a624d3b715903d"
server_key_iv: "56bdf0d5a593dd886fa00f8623ffad5f"
server_key_version: "kuduclusterkey@0"
I20260810 23:28:31.996232 2955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:28:32.015241 2955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:28:32.019553 2955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:28:32.019860 2955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:28:32.020856 2955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:28:32.022339 2955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:28:32.022415 2955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:32.022531 2955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:28:32.022579 2955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:32.056672 2955 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.67:43901
I20260810 23:28:32.056723 3163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.67:43901 every 8 connection(s)
I20260810 23:28:32.058071 2955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
I20260810 23:28:32.064498 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 2955
I20260810 23:28:32.064718 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance
I20260810 23:28:32.065301 15037 external_mini_cluster.cc:1428] Setting key cf2716b4b78091cbfa8c0ef99d3fba17
Aug 10 23:28:32 dist-test-slave-rmhg krb5kdc[1562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404511, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:28:32.080051 3168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:32.060886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:59517 (local address 127.14.175.126:34467)
0810 23:28:32.061434 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:32.061439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:32.062459 (+ 1020us) server_negotiation.cc:407] Connection header received
0810 23:28:32.064445 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:32.064452 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:32.064598 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:32.064788 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:32.066691 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:32.067483 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:32.069259 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:32.069730 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:32.069843 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:32.073827 (+ 3984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:32.073834 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:32.073837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:32.073874 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:32.076317 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:32.077370 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:32.077375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:32.077378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:32.077520 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:32.078185 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:32.078190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:32.078192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:32.078506 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:32.078649 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:32.079426 (+ 777us) server_negotiation.cc:299] Negotiation successful
0810 23:28:32.079693 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":276,"threads_started":1}
I20260810 23:28:32.081142 3169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:32.061544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:34467 (local address 127.14.175.67:59517)
0810 23:28:32.062265 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:32.062321 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:32.064138 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:32.065037 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:32.065046 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:32.065567 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:28:32.066497 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:32.066526 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:32.067646 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:32.067651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:32.069107 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:32.069121 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:32.069961 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:32.069970 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:32.070242 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:32.070879 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:28:32.070904 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:28:32.073548 (+ 2644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:28:32.076475 (+ 2927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:32.076486 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:32.076504 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:32.077238 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:32.077646 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:32.077652 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:32.077656 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:32.078048 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:32.078649 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:28:32.078659 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:28:32.079034 (+ 375us) client_negotiation.cc:769] Sending connection context
0810 23:28:32.079296 (+ 262us) client_negotiation.cc:240] Negotiation successful
0810 23:28:32.079886 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":270,"threads_started":1}
I20260810 23:28:32.082665 3164 heartbeater.cc:347] Connected to a master server at 127.14.175.126:34467
I20260810 23:28:32.082973 3164 heartbeater.cc:464] Registering TS with master...
I20260810 23:28:32.083731 3164 heartbeater.cc:511] Master 127.14.175.126:34467 requested a full tablet report, sending...
I20260810 23:28:32.085337 2347 ts_manager.cc:194] Registered new tserver with Master: aab0adaa90ca48bdb5b49c981a74e557 (127.14.175.67:43901)
I20260810 23:28:32.086202 2347 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.67@KRBTEST.COM'} at 127.14.175.67:59517
I20260810 23:28:32.094090 15037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:28:32.095863 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 2477
Aug 10 23:28:32 dist-test-slave-rmhg krb5kdc[1562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.65@KRBTEST.COM
I20260810 23:28:32.114121 3171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:32.096991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53680 (local address 127.14.175.65:36571)
0810 23:28:32.097713 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:32.097720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:32.097749 (+ 29us) server_negotiation.cc:407] Connection header received
0810 23:28:32.097818 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:32.097825 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:32.098064 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:32.098436 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:32.100060 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:32.101345 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:32.102868 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:32.103493 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:32.103601 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:32.107950 (+ 4349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:32.107968 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:32.107986 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:32.108030 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:32.111030 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:32.111865 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:32.111873 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:32.111876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:32.112104 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:32.112444 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:32.112448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:32.112450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:32.112818 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:32.113067 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:32.113434 (+ 367us) server_negotiation.cc:299] Negotiation successful
0810 23:28:32.113737 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":397,"threads_started":1}
I20260810 23:28:32.115486 2695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:28:32.202174 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 2477
I20260810 23:28:32.223096 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 2715
Aug 10 23:28:32 dist-test-slave-rmhg krb5kdc[1562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.66@KRBTEST.COM
I20260810 23:28:32.244745 3173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:32.224032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37810 (local address 127.14.175.66:36905)
0810 23:28:32.224855 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:32.224864 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:32.224894 (+ 30us) server_negotiation.cc:407] Connection header received
0810 23:28:32.224958 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:32.224964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:32.225254 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:32.225406 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:32.226721 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:32.227787 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:32.229342 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:32.229970 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:32.230080 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:32.238963 (+ 8883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:32.238982 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:32.239001 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:32.239073 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:32.242026 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:32.242789 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:32.242796 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:32.242799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:32.242954 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:32.243255 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:32.243258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:32.243260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:32.243625 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:32.243860 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:32.244109 (+ 249us) server_negotiation.cc:299] Negotiation successful
0810 23:28:32.244362 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":531,"threads_started":1}
I20260810 23:28:32.246122 2941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:28:32.328612 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 2715
I20260810 23:28:32.354781 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 2955
Aug 10 23:28:32 dist-test-slave-rmhg krb5kdc[1562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.67@KRBTEST.COM
I20260810 23:28:32.372438 3175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:32.355874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44078 (local address 127.14.175.67:43901)
0810 23:28:32.356510 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:32.356517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:32.356548 (+ 31us) server_negotiation.cc:407] Connection header received
0810 23:28:32.356613 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:32.356621 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:32.356847 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:32.356994 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:32.358392 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:32.359516 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:32.360853 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:32.361526 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:32.361651 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:32.366305 (+ 4654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:32.366321 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:32.366338 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:32.366379 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:32.369303 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:32.370171 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:32.370179 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:32.370181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:32.370324 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:32.370729 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:32.370733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:32.370735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:32.371131 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:32.371460 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:32.371789 (+ 329us) server_negotiation.cc:299] Negotiation successful
0810 23:28:32.372071 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":328,"threads_started":1}
I20260810 23:28:32.373909 3157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:28:32.473745 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 2955
I20260810 23:28:32.494881 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 2288
I20260810 23:28:32.496266 2391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:28:32.604095 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 2288
I20260810 23:28:32.632706 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:34467
--webserver_interface=127.14.175.126
--webserver_port=38567
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:38295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:34467
--ranger_config_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.175.84:37605/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.175.84:37605/kms with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:28:32.984869 3179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:28:32.985234 3179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:28:32.985350 3179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:28:32.992928 3179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:28:32.993069 3179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:28:32.993115 3179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:28:32.993209 3179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:28:32.993269 3179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:28:33.004094 3179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:38295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.175.84:37605/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:34467
--ranger_config_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:34467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=38567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:28:33.005585 3179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:28:33.007002 3179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:28:33.019310 3188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:33.019273 3186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:33.019228 3185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:28:33.020200 3179 server_base.cc:1034] running on GCE node
I20260810 23:28:33.021083 3179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:28:33.022681 3179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:28:33.023921 3179 hybrid_clock.cc:648] HybridClock initialized: now 1786404513023859 us; error 53 us; skew 500 ppm
I20260810 23:28:33.024305 3179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:28:33 dist-test-slave-rmhg krb5kdc[1562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404513, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:28:33.027558 3179 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:28:33.029114 3179 webserver.cc:460] Webserver started at http://127.14.175.126:38567/ using document root <none> and password file <none>
I20260810 23:28:33.029829 3179 fs_manager.cc:359] Metadata directory not provided
I20260810 23:28:33.029961 3179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 23:28:33 dist-test-slave-rmhg krb5kdc[1562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404513, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for HTTP/127.14.175.84@KRBTEST.COM
I20260810 23:28:33.110883 3179 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.001s
I20260810 23:28:33.114261 3205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:33.115787 3179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:28:33.115986 3179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "ff261d817152432bb0581bfb48106ed3"
format_stamp: "Formatted at 2026-08-10 23:28:28 on dist-test-slave-rmhg"
server_key: "8f7d778a29c893684e787ad1acfbb9d9"
server_key_iv: "99a8b347e5eecc76eb3d9c7edea5428c"
server_key_version: "kuduclusterkey@0"
I20260810 23:28:33.117105 3179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:28:33.135855 3179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:28:33.140290 3179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:28:33.140622 3179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:28:33.159672 3179 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:34467
I20260810 23:28:33.159711 3290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:34467 every 8 connection(s)
I20260810 23:28:33.161226 3179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:28:33.161881 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 3179
I20260810 23:28:33.162868 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.175.84:37605/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.65:36571
--local_ip_for_outbound_sockets=127.14.175.65
--tserver_master_addrs=127.14.175.126:34467
--webserver_port=33365
--webserver_interface=127.14.175.65
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:38295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.175.84:37605/kms with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:28:33.169029 3291 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:28:33.172887 3291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3: Bootstrap starting.
I20260810 23:28:33.199337 3291 log.cc:824] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3: Log is configured to *not* fsync() on all Append() calls
I20260810 23:28:33.209295 3291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:28:33.209846 3291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3: Bootstrap complete.
I20260810 23:28:33.219274 3291 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff261d817152432bb0581bfb48106ed3" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34467 } }
I20260810 23:28:33.220196 3291 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff261d817152432bb0581bfb48106ed3, State: Initialized, Role: FOLLOWER
I20260810 23:28:33.220940 3291 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff261d817152432bb0581bfb48106ed3" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34467 } }
I20260810 23:28:33.221196 3291 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:28:33.221326 3291 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:28:33.221519 3291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:28:33.225515 3291 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff261d817152432bb0581bfb48106ed3" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34467 } }
I20260810 23:28:33.226022 3291 leader_election.cc:302] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff261d817152432bb0581bfb48106ed3; no voters:
I20260810 23:28:33.226647 3291 leader_election.cc:288] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 23:28:33.226866 3302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 23:28:33.227847 3302 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [term 2 LEADER]: Becoming Leader. State: Replica: ff261d817152432bb0581bfb48106ed3, State: Running, Role: LEADER
I20260810 23:28:33.228587 3302 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff261d817152432bb0581bfb48106ed3" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34467 } }
I20260810 23:28:33.228861 3291 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:28:33.231247 3303 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ff261d817152432bb0581bfb48106ed3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff261d817152432bb0581bfb48106ed3" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34467 } } }
I20260810 23:28:33.231623 3303 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [sys.catalog]: This master's current role is: LEADER
I20260810 23:28:33.235914 3306 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff261d817152432bb0581bfb48106ed3. Latest consensus state: current_term: 2 leader_uuid: "ff261d817152432bb0581bfb48106ed3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff261d817152432bb0581bfb48106ed3" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34467 } } }
I20260810 23:28:33.236366 3306 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3 [sys.catalog]: This master's current role is: LEADER
I20260810 23:28:33.237421 3310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:28:33.244544 3310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:28:33.246354 3310 catalog_manager.cc:1259] Loaded cluster ID: 73d684310dc84ad2890fa69730791bc2
I20260810 23:28:33.246565 3310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:28:33.251184 3310 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:28:33.252930 3310 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3: Loaded TSK: 0
I20260810 23:28:33.254763 3310 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:28:33.257995 3291 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260810 23:28:33.647342 3295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:28:33.647810 3295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:28:33.648038 3295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:28:33.659765 3295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:28:33.660002 3295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:28:33.660221 3295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.65
I20260810 23:28:33.676354 3295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:38295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.175.84:37605/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.65:36571
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.175.65
--webserver_port=33365
--tserver_master_addrs=127.14.175.126:34467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.65
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:28:33.678531 3295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:28:33.680640 3295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:28:33.713848 3358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:33.715902 3359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:33.720440 3363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:28:33.721716 3295 server_base.cc:1034] running on GCE node
I20260810 23:28:33.722678 3295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:28:33.730195 3295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:28:33.731660 3295 hybrid_clock.cc:648] HybridClock initialized: now 1786404513731555 us; error 123 us; skew 500 ppm
I20260810 23:28:33.732105 3295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:28:33 dist-test-slave-rmhg krb5kdc[1562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404513, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:28:33.736452 3295 init.cc:377] Logged in from keytab as kudu/127.14.175.65@KRBTEST.COM (short username kudu)
I20260810 23:28:33.746876 3295 webserver.cc:460] Webserver started at http://127.14.175.65:33365/ using document root <none> and password file <none>
I20260810 23:28:33.747921 3295 fs_manager.cc:359] Metadata directory not provided
I20260810 23:28:33.748219 3295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 23:28:33 dist-test-slave-rmhg krb5kdc[1562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404513, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for HTTP/127.14.175.84@KRBTEST.COM
I20260810 23:28:33.785470 3295 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.006s sys 0.000s
I20260810 23:28:33.789747 3374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:33.791872 3295 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:28:33.792199 3295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-0/wal
uuid: "04e6d9f68e874b24b86fa0750afbed0b"
format_stamp: "Formatted at 2026-08-10 23:28:30 on dist-test-slave-rmhg"
server_key: "9ef4cadbd9673b49be9e9a6066533b3f"
server_key_iv: "257b0df88c0128ced7365d8bcee5c93e"
server_key_version: "kuduclusterkey@0"
I20260810 23:28:33.793895 3295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:28:33.848804 3295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:28:33.853538 3295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:28:33.853948 3295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:28:33.855342 3295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:28:33.857123 3295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:28:33.857306 3295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260810 23:28:33.857452 3295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:28:33.857533 3295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:33.928360 3582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.65:36571 every 8 connection(s)
I20260810 23:28:33.928815 3295 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.65:36571
I20260810 23:28:33.930670 3295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
I20260810 23:28:33.938565 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 3295
I20260810 23:28:33.939510 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.175.84:37605/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.66:36905
--local_ip_for_outbound_sockets=127.14.175.66
--tserver_master_addrs=127.14.175.126:34467
--webserver_port=36097
--webserver_interface=127.14.175.66
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:38295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.175.84:37605/kms with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:28:33 dist-test-slave-rmhg krb5kdc[1562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404513, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:28:33.996896 3589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:33.952127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:34467 (local address 127.14.175.65:36589)
0810 23:28:33.967361 (+ 15234us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:33.967437 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:33.969894 (+ 2457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:33.972346 (+ 2452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:33.972360 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:33.973007 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:28:33.973927 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:33.973956 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:33.976530 (+ 2574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:33.976536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:33.978712 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:33.978732 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:33.980358 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:33.980367 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:33.980672 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:33.982867 (+ 2195us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:28:33.982906 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:28:33.985890 (+ 2984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:28:33.990956 (+ 5066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:33.990967 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:33.990988 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:33.991840 (+ 852us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:33.992525 (+ 685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:33.992530 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:33.992534 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:33.992823 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:33.993977 (+ 1154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:28:33.993987 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:28:33.994431 (+ 444us) client_negotiation.cc:769] Sending connection context
0810 23:28:33.994766 (+ 335us) client_negotiation.cc:240] Negotiation successful
0810 23:28:33.995382 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14911,"thread_start_us":14642,"threads_started":1}
I20260810 23:28:33.998483 3596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:33.953463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:36589 (local address 127.14.175.126:34467)
0810 23:28:33.966400 (+ 12937us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:33.966430 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:33.968009 (+ 1579us) server_negotiation.cc:407] Connection header received
0810 23:28:33.970777 (+ 2768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:33.970801 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:33.971409 (+ 608us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:33.971923 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:33.974195 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:33.976267 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:33.978940 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:33.980071 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:33.980290 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:33.986323 (+ 6033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:33.986337 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:33.986363 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:33.986435 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:33.990716 (+ 4281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:33.992093 (+ 1377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:33.992104 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:33.992114 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:33.992349 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:33.992975 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:33.992980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:33.992983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:33.993798 (+ 815us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:33.996305 (+ 2507us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:33.996642 (+ 337us) server_negotiation.cc:299] Negotiation successful
0810 23:28:33.997090 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12519,"thread_start_us":12867,"threads_started":1}
I20260810 23:28:34.002360 3585 heartbeater.cc:347] Connected to a master server at 127.14.175.126:34467
I20260810 23:28:34.002707 3585 heartbeater.cc:464] Registering TS with master...
I20260810 23:28:34.003630 3585 heartbeater.cc:511] Master 127.14.175.126:34467 requested a full tablet report, sending...
W20260810 23:28:34.007450 3585 heartbeater.cc:644] Failed to heartbeat to 127.14.175.126:34467 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 23:28:34.007732 3585 heartbeater.cc:464] Registering TS with master...
I20260810 23:28:34.008069 3585 heartbeater.cc:511] Master 127.14.175.126:34467 requested a full tablet report, sending...
I20260810 23:28:34.009681 3585 heartbeater.cc:464] Registering TS with master...
I20260810 23:28:34.010077 3585 heartbeater.cc:511] Master 127.14.175.126:34467 requested a full tablet report, sending...
W20260810 23:28:34.011461 3585 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 23:28:34.415729 3588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:28:34.416137 3588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:28:34.416285 3588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:28:34.428072 3588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:28:34.428238 3588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:28:34.428505 3588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.66
I20260810 23:28:34.444759 3588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:38295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.175.84:37605/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.66:36905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.175.66
--webserver_port=36097
--tserver_master_addrs=127.14.175.126:34467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.66
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
Aug 10 23:28:34 dist-test-slave-rmhg krb5kdc[1562](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786404514, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:28:34.446720 3588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:28:34.448655 3588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:28:34.478658 3617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:34.485632 3616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:34.499318 3619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:28:34.499792 3588 server_base.cc:1034] running on GCE node
I20260810 23:28:34.500686 3588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:28:34.502241 3588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:28:34.503690 3588 hybrid_clock.cc:648] HybridClock initialized: now 1786404514503392 us; error 271 us; skew 500 ppm
I20260810 23:28:34.504117 3588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:28:34 dist-test-slave-rmhg krb5kdc[1562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404514, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:28:34.511880 3588 init.cc:377] Logged in from keytab as kudu/127.14.175.66@KRBTEST.COM (short username kudu)
I20260810 23:28:34.514170 3588 webserver.cc:460] Webserver started at http://127.14.175.66:36097/ using document root <none> and password file <none>
I20260810 23:28:34.515161 3588 fs_manager.cc:359] Metadata directory not provided
I20260810 23:28:34.515306 3588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 23:28:34 dist-test-slave-rmhg krb5kdc[1562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404514, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for HTTP/127.14.175.84@KRBTEST.COM
I20260810 23:28:34.543852 3588 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260810 23:28:34.547756 3636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:34.549693 3588 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260810 23:28:34.549908 3588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-1/wal
uuid: "eea228eefebc4347acb2db74799297a4"
format_stamp: "Formatted at 2026-08-10 23:28:31 on dist-test-slave-rmhg"
server_key: "e52eaaca65aac94652a0e42dc9a346a1"
server_key_iv: "fbbb1bbad22074d398ca954e8281e3e0"
server_key_version: "kuduclusterkey@0"
I20260810 23:28:34.551357 3588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:28:34.595247 3588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:28:34.600769 3588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:28:34.601217 3588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:28:34.602584 3588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:28:34.611389 3588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:28:34.611536 3588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:34.611711 3588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:28:34.611794 3588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:34.671284 3588 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.66:36905
I20260810 23:28:34.671324 3818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.66:36905 every 8 connection(s)
I20260810 23:28:34.673029 3588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
I20260810 23:28:34.675792 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 3588
I20260810 23:28:34.676559 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.175.84:37605/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.67:43901
--local_ip_for_outbound_sockets=127.14.175.67
--tserver_master_addrs=127.14.175.126:34467
--webserver_port=41201
--webserver_interface=127.14.175.67
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:38295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.175.84:37605/kms with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:28:34 dist-test-slave-rmhg krb5kdc[1562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404514, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:28:34.709532 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:34.677831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:46737 (local address 127.14.175.126:34467)
0810 23:28:34.678324 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:34.678331 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:34.687219 (+ 8888us) server_negotiation.cc:407] Connection header received
0810 23:28:34.689137 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:34.689142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:34.689261 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:34.689371 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:34.691561 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:34.692645 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:34.695053 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:34.695692 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:34.695817 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:34.700455 (+ 4638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:34.700463 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:34.700468 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:34.700518 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:34.704129 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:34.705186 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:34.705191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:34.705195 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:34.705348 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:34.705973 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:34.705978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:34.705982 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:34.706357 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:34.706492 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:34.708850 (+ 2358us) server_negotiation.cc:299] Negotiation successful
0810 23:28:34.709129 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":255,"threads_started":1}
I20260810 23:28:34.709720 3822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:34.676929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:34467 (local address 127.14.175.66:46737)
0810 23:28:34.686305 (+ 9376us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:34.686368 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:34.688637 (+ 2269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:34.689891 (+ 1254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:34.689904 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:34.690556 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:28:34.691363 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:34.691383 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:34.692847 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:34.692853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:34.694860 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:34.694876 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:34.696145 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:34.696153 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:34.696455 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:34.697339 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:28:34.697371 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:28:34.700114 (+ 2743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:28:34.704335 (+ 4221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:34.704347 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:34.704369 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:34.705000 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:34.705478 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:34.705483 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:34.705486 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:34.705807 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:34.706591 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:28:34.706601 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:28:34.707097 (+ 496us) client_negotiation.cc:769] Sending connection context
0810 23:28:34.707436 (+ 339us) client_negotiation.cc:240] Negotiation successful
0810 23:28:34.708089 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9034,"thread_start_us":9184,"threads_started":1}
I20260810 23:28:34.711790 3820 heartbeater.cc:347] Connected to a master server at 127.14.175.126:34467
I20260810 23:28:34.712242 3820 heartbeater.cc:464] Registering TS with master...
I20260810 23:28:34.713089 3820 heartbeater.cc:511] Master 127.14.175.126:34467 requested a full tablet report, sending...
I20260810 23:28:34.715376 3229 ts_manager.cc:194] Registered new tserver with Master: eea228eefebc4347acb2db74799297a4 (127.14.175.66:36905)
I20260810 23:28:34.718004 3229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.66@KRBTEST.COM'} at 127.14.175.66:46737
Aug 10 23:28:34 dist-test-slave-rmhg krb5kdc[1562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404514, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for HTTP/127.14.175.84@KRBTEST.COM
I20260810 23:28:35.011548 3585 heartbeater.cc:464] Registering TS with master...
I20260810 23:28:35.012042 3585 heartbeater.cc:511] Master 127.14.175.126:34467 requested a full tablet report, sending...
I20260810 23:28:35.013375 3229 ts_manager.cc:194] Registered new tserver with Master: 04e6d9f68e874b24b86fa0750afbed0b (127.14.175.65:36571)
I20260810 23:28:35.014566 3229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.65@KRBTEST.COM'} at 127.14.175.65:36589
W20260810 23:28:35.160322 3825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:28:35.160816 3825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:28:35.161006 3825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:28:35.172617 3825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:28:35.172902 3825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:28:35.173187 3825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.67
Aug 10 23:28:35 dist-test-slave-rmhg krb5kdc[1562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404514, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for HTTP/127.14.175.84@KRBTEST.COM
I20260810 23:28:35.187446 3825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:38295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.175.84:37605/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.67:43901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.175.67
--webserver_port=41201
--tserver_master_addrs=127.14.175.126:34467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.67
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:28:35.188884 3825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:28:35.190863 3825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:28:35.262710 3838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:35.263449 3835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:35.269296 3844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:28:35.271265 3825 server_base.cc:1034] running on GCE node
I20260810 23:28:35.272233 3825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:28:35.282593 3825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:28:35.284185 3825 hybrid_clock.cc:648] HybridClock initialized: now 1786404515284106 us; error 95 us; skew 500 ppm
I20260810 23:28:35.284730 3825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:28:35 dist-test-slave-rmhg krb5kdc[1562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404515, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:28:35.289578 3825 init.cc:377] Logged in from keytab as kudu/127.14.175.67@KRBTEST.COM (short username kudu)
I20260810 23:28:35.318542 3825 webserver.cc:460] Webserver started at http://127.14.175.67:41201/ using document root <none> and password file <none>
I20260810 23:28:35.319741 3825 fs_manager.cc:359] Metadata directory not provided
I20260810 23:28:35.320041 3825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 23:28:35 dist-test-slave-rmhg krb5kdc[1562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404515, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for HTTP/127.14.175.84@KRBTEST.COM
I20260810 23:28:35.365413 3825 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.002s sys 0.009s
I20260810 23:28:35.373919 3853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:35.376624 3825 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.008s sys 0.000s
I20260810 23:28:35.376832 3825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-2/wal
uuid: "aab0adaa90ca48bdb5b49c981a74e557"
format_stamp: "Formatted at 2026-08-10 23:28:31 on dist-test-slave-rmhg"
server_key: "e50d3c9e9daabbe1d0a624d3b715903d"
server_key_iv: "56bdf0d5a593dd886fa00f8623ffad5f"
server_key_version: "kuduclusterkey@0"
I20260810 23:28:35.378379 3825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:28:35.411669 3825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:28:35.416972 3825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:28:35.417379 3825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:28:35.421706 3825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:28:35.423796 3825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:28:35.423919 3825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:35.424021 3825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:28:35.424094 3825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:35.472427 3825 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.67:43901
I20260810 23:28:35.472461 4035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.67:43901 every 8 connection(s)
I20260810 23:28:35.474347 3825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
I20260810 23:28:35.477576 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 3825
Aug 10 23:28:35 dist-test-slave-rmhg krb5kdc[1562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404515, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:28:35.515648 4039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:35.479119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42700 (local address 127.14.175.126:34467)
0810 23:28:35.497971 (+ 18852us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:35.497977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:35.497992 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:28:35.498589 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:35.498596 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:35.498712 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:35.498833 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:35.500976 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:35.501940 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:35.503986 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:35.504748 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:35.504864 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:35.508348 (+ 3484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:35.508354 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:35.508359 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:35.508417 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:35.512194 (+ 3777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:35.513130 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:35.513134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:35.513137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:35.513322 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:35.513770 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:35.513776 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:35.513779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:35.514200 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:35.514493 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:35.514817 (+ 324us) server_negotiation.cc:299] Negotiation successful
0810 23:28:35.515065 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":18741,"thread_start_us":17731,"threads_started":1}
I20260810 23:28:35.517802 4046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:35.477313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:35349 (local address 127.14.175.126:34467)
0810 23:28:35.497112 (+ 19799us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:35.497120 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:35.497822 (+ 702us) server_negotiation.cc:407] Connection header received
0810 23:28:35.497920 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:35.497925 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:35.498031 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:35.498139 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:35.500369 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:35.501814 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:35.503467 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:35.503880 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:35.503986 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:35.507995 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:35.507999 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:35.508002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:35.508049 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:35.512256 (+ 4207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:35.513097 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:35.513103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:35.513106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:35.513315 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:35.513859 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:35.513863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:35.513866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:35.514211 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:35.514544 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:35.517315 (+ 2771us) server_negotiation.cc:299] Negotiation successful
0810 23:28:35.517520 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":189,"server-negotiator.queue_time_us":19652,"spinlock_wait_cycles":4864,"thread_start_us":20368,"threads_started":1}
I20260810 23:28:35.518050 4040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:35.477973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:34467 (local address 127.14.175.67:35349)
0810 23:28:35.492537 (+ 14564us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:35.492634 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:35.495301 (+ 2667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:35.498510 (+ 3209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:35.498525 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:35.499192 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:28:35.500135 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:35.500166 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:35.501992 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:35.501998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:35.503319 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:35.503332 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:35.504188 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:35.504197 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:35.504508 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:35.505689 (+ 1181us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:28:35.505730 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:28:35.507756 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:28:35.512454 (+ 4698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:35.512466 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:35.512484 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:35.512961 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:35.513466 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:35.513471 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:35.513474 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:35.513722 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:35.514387 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:28:35.514395 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:28:35.514901 (+ 506us) client_negotiation.cc:769] Sending connection context
0810 23:28:35.515236 (+ 335us) client_negotiation.cc:240] Negotiation successful
0810 23:28:35.515846 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14151,"thread_start_us":17730,"threads_started":1}
I20260810 23:28:35.519485 4037 heartbeater.cc:347] Connected to a master server at 127.14.175.126:34467
I20260810 23:28:35.519845 4037 heartbeater.cc:464] Registering TS with master...
I20260810 23:28:35.520843 4037 heartbeater.cc:511] Master 127.14.175.126:34467 requested a full tablet report, sending...
I20260810 23:28:35.522589 3229 ts_manager.cc:194] Registered new tserver with Master: aab0adaa90ca48bdb5b49c981a74e557 (127.14.175.67:43901)
I20260810 23:28:35.523741 3229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.67@KRBTEST.COM'} at 127.14.175.67:35349
I20260810 23:28:35.531800 15037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:28:35.548017 4046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:35.535160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42708 (local address 127.14.175.126:34467)
0810 23:28:35.535343 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:35.535347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:35.535758 (+ 411us) server_negotiation.cc:407] Connection header received
0810 23:28:35.535900 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:35.535903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:35.535982 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:35.536067 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:35.537647 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:35.538416 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:35.539878 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:35.540323 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:35.540427 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:35.542991 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:35.542996 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:35.543000 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:35.543035 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:35.545677 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:35.546296 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:35.546301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:35.546303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:35.546452 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:35.546763 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:35.546766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:35.546768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:35.547070 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:35.547259 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:35.547583 (+ 324us) server_negotiation.cc:299] Negotiation successful
0810 23:28:35.547771 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 23:28:35.553079 3229 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:28:35.560601 3229 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:28:35.593405 4066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:35.576471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37812 (local address 127.14.175.66:36905)
0810 23:28:35.581640 (+ 5169us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:35.581652 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:35.582318 (+ 666us) server_negotiation.cc:407] Connection header received
0810 23:28:35.582439 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:35.582447 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:35.582747 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:35.582956 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:28:35.583786 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:35.585780 (+ 1994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:35.590075 (+ 4295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:35.591081 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:35.591222 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:35.591456 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:35.592886 (+ 1430us) server_negotiation.cc:299] Negotiation successful
0810 23:28:35.593075 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4987,"thread_start_us":4886,"threads_started":1}
I20260810 23:28:35.594039 4061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:35.575773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:36905 (local address 127.0.0.1:37812)
0810 23:28:35.581986 (+ 6213us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:35.582006 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:35.582172 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:35.583259 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:35.583264 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:35.583290 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:28:35.583618 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:35.583631 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:35.585967 (+ 2336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:35.585974 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:35.589909 (+ 3935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:35.589920 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:35.591778 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:35.591790 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:35.592044 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:35.592227 (+ 183us) client_negotiation.cc:769] Sending connection context
0810 23:28:35.592390 (+ 163us) client_negotiation.cc:240] Negotiation successful
0810 23:28:35.592613 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6084,"mutex_wait_us":82,"thread_start_us":849,"threads_started":1}
I20260810 23:28:35.596516 4064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:35.575423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:43901 (local address 127.0.0.1:44086)
0810 23:28:35.576779 (+ 1356us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:35.576820 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:35.576983 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:35.589128 (+ 12145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:35.589133 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:35.589189 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:28:35.589641 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:35.589652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:35.591575 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:35.591580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:35.593732 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:35.593748 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:35.595665 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:35.595672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:35.595868 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:35.595894 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 23:28:35.595976 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 23:28:35.596119 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1068,"mutex_wait_us":26,"thread_start_us":6310,"threads_started":1}
I20260810 23:28:35.596561 4063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:35.575532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44086 (local address 127.14.175.67:43901)
0810 23:28:35.588535 (+ 13003us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:35.588542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:35.588570 (+ 28us) server_negotiation.cc:407] Connection header received
0810 23:28:35.588656 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:35.588663 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:35.588900 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:35.589118 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:28:35.589868 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:35.591448 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:35.593909 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:35.595390 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:35.595572 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:35.595805 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:35.596040 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 23:28:35.596217 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12831,"thread_start_us":12688,"threads_started":1}
I20260810 23:28:35.596774 4068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:35.582663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53690 (local address 127.14.175.65:36571)
0810 23:28:35.588039 (+ 5376us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:35.588047 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:35.588080 (+ 33us) server_negotiation.cc:407] Connection header received
0810 23:28:35.588209 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:35.588218 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:35.588501 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:35.588752 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:28:35.589518 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:35.591131 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:35.593508 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:35.595023 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:35.595216 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:35.595461 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:35.596279 (+ 818us) server_negotiation.cc:299] Negotiation successful
0810 23:28:35.596453 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5198,"thread_start_us":5027,"threads_started":1}
I20260810 23:28:35.596150 4069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:35.583893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:36571 (local address 127.0.0.1:53690)
0810 23:28:35.587793 (+ 3900us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:35.587814 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:35.587938 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:35.588743 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:35.588748 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:35.588777 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:28:35.589298 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:35.589314 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:35.591306 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:35.591312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:35.593292 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:35.593306 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:35.595312 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:35.595318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:35.595487 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:35.595520 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 23:28:35.595613 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 23:28:35.595755 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3740,"mutex_wait_us":7,"thread_start_us":3531,"threads_started":1}
I20260810 23:28:35.600404 3942 tablet_service.cc:1467] Processing CreateTablet for tablet cce84d138b0542f1a61ca442f9a625f3 (DEFAULT_TABLE table=test-table [id=ed4775eff511494db72bb66eca5bc16d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:28:35.600390 3462 tablet_service.cc:1467] Processing CreateTablet for tablet cce84d138b0542f1a61ca442f9a625f3 (DEFAULT_TABLE table=test-table [id=ed4775eff511494db72bb66eca5bc16d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:28:35.601722 3942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cce84d138b0542f1a61ca442f9a625f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:28:35.601836 3462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cce84d138b0542f1a61ca442f9a625f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:28:35.601421 3718 tablet_service.cc:1467] Processing CreateTablet for tablet cce84d138b0542f1a61ca442f9a625f3 (DEFAULT_TABLE table=test-table [id=ed4775eff511494db72bb66eca5bc16d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:28:35.602689 3718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cce84d138b0542f1a61ca442f9a625f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:28:35.616314 4070 tablet_bootstrap.cc:492] T cce84d138b0542f1a61ca442f9a625f3 P 04e6d9f68e874b24b86fa0750afbed0b: Bootstrap starting.
I20260810 23:28:35.619746 4070 tablet_bootstrap.cc:654] T cce84d138b0542f1a61ca442f9a625f3 P 04e6d9f68e874b24b86fa0750afbed0b: Neither blocks nor log segments found. Creating new log.
I20260810 23:28:35.620496 4073 tablet_bootstrap.cc:492] T cce84d138b0542f1a61ca442f9a625f3 P aab0adaa90ca48bdb5b49c981a74e557: Bootstrap starting.
I20260810 23:28:35.621366 4070 log.cc:824] T cce84d138b0542f1a61ca442f9a625f3 P 04e6d9f68e874b24b86fa0750afbed0b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:28:35.622272 4075 tablet_bootstrap.cc:492] T cce84d138b0542f1a61ca442f9a625f3 P eea228eefebc4347acb2db74799297a4: Bootstrap starting.
I20260810 23:28:35.624315 4073 tablet_bootstrap.cc:654] T cce84d138b0542f1a61ca442f9a625f3 P aab0adaa90ca48bdb5b49c981a74e557: Neither blocks nor log segments found. Creating new log.
I20260810 23:28:35.625640 4075 tablet_bootstrap.cc:654] T cce84d138b0542f1a61ca442f9a625f3 P eea228eefebc4347acb2db74799297a4: Neither blocks nor log segments found. Creating new log.
I20260810 23:28:35.625730 4073 log.cc:824] T cce84d138b0542f1a61ca442f9a625f3 P aab0adaa90ca48bdb5b49c981a74e557: Log is configured to *not* fsync() on all Append() calls
I20260810 23:28:35.626441 4070 tablet_bootstrap.cc:492] T cce84d138b0542f1a61ca442f9a625f3 P 04e6d9f68e874b24b86fa0750afbed0b: No bootstrap required, opened a new log
I20260810 23:28:35.626843 4070 ts_tablet_manager.cc:1397] T cce84d138b0542f1a61ca442f9a625f3 P 04e6d9f68e874b24b86fa0750afbed0b: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260810 23:28:35.626875 4075 log.cc:824] T cce84d138b0542f1a61ca442f9a625f3 P eea228eefebc4347acb2db74799297a4: Log is configured to *not* fsync() on all Append() calls
I20260810 23:28:35.629575 4073 tablet_bootstrap.cc:492] T cce84d138b0542f1a61ca442f9a625f3 P aab0adaa90ca48bdb5b49c981a74e557: No bootstrap required, opened a new log
I20260810 23:28:35.629904 4073 ts_tablet_manager.cc:1397] T cce84d138b0542f1a61ca442f9a625f3 P aab0adaa90ca48bdb5b49c981a74e557: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260810 23:28:35.630188 4075 tablet_bootstrap.cc:492] T cce84d138b0542f1a61ca442f9a625f3 P eea228eefebc4347acb2db74799297a4: No bootstrap required, opened a new log
I20260810 23:28:35.630486 4075 ts_tablet_manager.cc:1397] T cce84d138b0542f1a61ca442f9a625f3 P eea228eefebc4347acb2db74799297a4: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 23:28:35.634829 4070 raft_consensus.cc:348] T cce84d138b0542f1a61ca442f9a625f3 P 04e6d9f68e874b24b86fa0750afbed0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab0adaa90ca48bdb5b49c981a74e557" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43901 } } peers { permanent_uuid: "04e6d9f68e874b24b86fa0750afbed0b" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 36571 } } peers { permanent_uuid: "eea228eefebc4347acb2db74799297a4" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36905 } }
I20260810 23:28:35.635210 4070 raft_consensus.cc:374] T cce84d138b0542f1a61ca442f9a625f3 P 04e6d9f68e874b24b86fa0750afbed0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:28:35.635352 4070 raft_consensus.cc:729] T cce84d138b0542f1a61ca442f9a625f3 P 04e6d9f68e874b24b86fa0750afbed0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e6d9f68e874b24b86fa0750afbed0b, State: Initialized, Role: FOLLOWER
I20260810 23:28:35.635957 4070 consensus_queue.cc:260] T cce84d138b0542f1a61ca442f9a625f3 P 04e6d9f68e874b24b86fa0750afbed0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab0adaa90ca48bdb5b49c981a74e557" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43901 } } peers { permanent_uuid: "04e6d9f68e874b24b86fa0750afbed0b" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 36571 } } peers { permanent_uuid: "eea228eefebc4347acb2db74799297a4" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36905 } }
I20260810 23:28:35.637190 3585 heartbeater.cc:503] Master 127.14.175.126:34467 was elected leader, sending a full tablet report...
I20260810 23:28:35.637923 4070 ts_tablet_manager.cc:1428] T cce84d138b0542f1a61ca442f9a625f3 P 04e6d9f68e874b24b86fa0750afbed0b: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260810 23:28:35.640847 4075 raft_consensus.cc:348] T cce84d138b0542f1a61ca442f9a625f3 P eea228eefebc4347acb2db74799297a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab0adaa90ca48bdb5b49c981a74e557" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43901 } } peers { permanent_uuid: "04e6d9f68e874b24b86fa0750afbed0b" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 36571 } } peers { permanent_uuid: "eea228eefebc4347acb2db74799297a4" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36905 } }
I20260810 23:28:35.641459 4075 raft_consensus.cc:374] T cce84d138b0542f1a61ca442f9a625f3 P eea228eefebc4347acb2db74799297a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:28:35.641589 4075 raft_consensus.cc:729] T cce84d138b0542f1a61ca442f9a625f3 P eea228eefebc4347acb2db74799297a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eea228eefebc4347acb2db74799297a4, State: Initialized, Role: FOLLOWER
I20260810 23:28:35.641559 4073 raft_consensus.cc:348] T cce84d138b0542f1a61ca442f9a625f3 P aab0adaa90ca48bdb5b49c981a74e557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab0adaa90ca48bdb5b49c981a74e557" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43901 } } peers { permanent_uuid: "04e6d9f68e874b24b86fa0750afbed0b" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 36571 } } peers { permanent_uuid: "eea228eefebc4347acb2db74799297a4" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36905 } }
I20260810 23:28:35.642045 4073 raft_consensus.cc:374] T cce84d138b0542f1a61ca442f9a625f3 P aab0adaa90ca48bdb5b49c981a74e557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:28:35.642153 4073 raft_consensus.cc:729] T cce84d138b0542f1a61ca442f9a625f3 P aab0adaa90ca48bdb5b49c981a74e557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aab0adaa90ca48bdb5b49c981a74e557, State: Initialized, Role: FOLLOWER
I20260810 23:28:35.642271 4075 consensus_queue.cc:260] T cce84d138b0542f1a61ca442f9a625f3 P eea228eefebc4347acb2db74799297a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab0adaa90ca48bdb5b49c981a74e557" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43901 } } peers { permanent_uuid: "04e6d9f68e874b24b86fa0750afbed0b" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 36571 } } peers { permanent_uuid: "eea228eefebc4347acb2db74799297a4" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36905 } }
I20260810 23:28:35.642763 4073 consensus_queue.cc:260] T cce84d138b0542f1a61ca442f9a625f3 P aab0adaa90ca48bdb5b49c981a74e557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab0adaa90ca48bdb5b49c981a74e557" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43901 } } peers { permanent_uuid: "04e6d9f68e874b24b86fa0750afbed0b" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 36571 } } peers { permanent_uuid: "eea228eefebc4347acb2db74799297a4" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36905 } }
I20260810 23:28:35.643543 3820 heartbeater.cc:503] Master 127.14.175.126:34467 was elected leader, sending a full tablet report...
I20260810 23:28:35.644344 4075 ts_tablet_manager.cc:1428] T cce84d138b0542f1a61ca442f9a625f3 P eea228eefebc4347acb2db74799297a4: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260810 23:28:35.644577 4037 heartbeater.cc:503] Master 127.14.175.126:34467 was elected leader, sending a full tablet report...
I20260810 23:28:35.645272 4073 ts_tablet_manager.cc:1428] T cce84d138b0542f1a61ca442f9a625f3 P aab0adaa90ca48bdb5b49c981a74e557: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
W20260810 23:28:35.681792 3821 tablet.cc:2369] T cce84d138b0542f1a61ca442f9a625f3 P eea228eefebc4347acb2db74799297a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:28:35.695956 3586 tablet.cc:2369] T cce84d138b0542f1a61ca442f9a625f3 P 04e6d9f68e874b24b86fa0750afbed0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:28:35.726186 4038 tablet.cc:2369] T cce84d138b0542f1a61ca442f9a625f3 P aab0adaa90ca48bdb5b49c981a74e557: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:28:35.792711 4084 raft_consensus.cc:482] T cce84d138b0542f1a61ca442f9a625f3 P 04e6d9f68e874b24b86fa0750afbed0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:28:35.793013 4084 raft_consensus.cc:504] T cce84d138b0542f1a61ca442f9a625f3 P 04e6d9f68e874b24b86fa0750afbed0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab0adaa90ca48bdb5b49c981a74e557" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43901 } } peers { permanent_uuid: "04e6d9f68e874b24b86fa0750afbed0b" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 36571 } } peers { permanent_uuid: "eea228eefebc4347acb2db74799297a4" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36905 } }
I20260810 23:28:35.794903 4084 leader_election.cc:288] T cce84d138b0542f1a61ca442f9a625f3 P 04e6d9f68e874b24b86fa0750afbed0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aab0adaa90ca48bdb5b49c981a74e557 (127.14.175.67:43901), eea228eefebc4347acb2db74799297a4 (127.14.175.66:36905)
I20260810 23:28:35.802735 4063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:35.795448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:54725 (local address 127.14.175.67:43901)
0810 23:28:35.795676 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:35.795682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:35.796378 (+ 696us) server_negotiation.cc:407] Connection header received
0810 23:28:35.796583 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:35.796587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:35.796653 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:35.796733 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:28:35.797457 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:35.798595 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:35.800820 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:35.801685 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:35.801852 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:35.801939 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:35.802288 (+ 349us) server_negotiation.cc:299] Negotiation successful
0810 23:28:35.802446 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 23:28:35.802742 4066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:35.795554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:49221 (local address 127.14.175.66:36905)
0810 23:28:35.795784 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:35.795790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:35.796434 (+ 644us) server_negotiation.cc:407] Connection header received
0810 23:28:35.796638 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:35.796642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:35.796705 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:35.796783 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:28:35.797631 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:35.798669 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:35.800806 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:35.801709 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:35.801880 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:35.801957 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:35.802314 (+ 357us) server_negotiation.cc:299] Negotiation successful
0810 23:28:35.802458 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 23:28:35.802940 4090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:35.795469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:36905 (local address 127.14.175.65:49221)
0810 23:28:35.796317 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:35.796332 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:35.796467 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:35.796820 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:35.796824 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:35.796848 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:28:35.797452 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:35.797468 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:35.798807 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:35.798811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:35.800622 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:35.800641 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:35.802001 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:35.802007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:35.802147 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:35.802171 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 23:28:35.802263 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 23:28:35.802446 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"mutex_wait_us":44,"thread_start_us":423,"threads_started":1}
I20260810 23:28:35.802970 4089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:35.795162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:43901 (local address 127.14.175.65:54725)
0810 23:28:35.796239 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:35.796261 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:35.796434 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:35.796771 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:35.796775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:35.796800 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:28:35.797256 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:35.797271 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:35.798804 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:35.798809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:35.800648 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:35.800662 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:35.801945 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:35.801952 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:35.802120 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:35.802147 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 23:28:35.802229 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 23:28:35.802454 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":930,"mutex_wait_us":147,"thread_start_us":685,"threads_started":1}
I20260810 23:28:35.804159 3751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cce84d138b0542f1a61ca442f9a625f3" candidate_uuid: "04e6d9f68e874b24b86fa0750afbed0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eea228eefebc4347acb2db74799297a4" is_pre_election: true
I20260810 23:28:35.804207 3975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cce84d138b0542f1a61ca442f9a625f3" candidate_uuid: "04e6d9f68e874b24b86fa0750afbed0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aab0adaa90ca48bdb5b49c981a74e557" is_pre_election: true
I20260810 23:28:35.804720 3975 raft_consensus.cc:2454] T cce84d138b0542f1a61ca442f9a625f3 P aab0adaa90ca48bdb5b49c981a74e557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04e6d9f68e874b24b86fa0750afbed0b in term 0.
I20260810 23:28:35.804721 3751 raft_consensus.cc:2454] T cce84d138b0542f1a61ca442f9a625f3 P eea228eefebc4347acb2db74799297a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04e6d9f68e874b24b86fa0750afbed0b in term 0.
I20260810 23:28:35.805464 3379 leader_election.cc:302] T cce84d138b0542f1a61ca442f9a625f3 P 04e6d9f68e874b24b86fa0750afbed0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04e6d9f68e874b24b86fa0750afbed0b, aab0adaa90ca48bdb5b49c981a74e557; no voters:
I20260810 23:28:35.805908 4084 raft_consensus.cc:2781] T cce84d138b0542f1a61ca442f9a625f3 P 04e6d9f68e874b24b86fa0750afbed0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:28:35.806082 4084 raft_consensus.cc:482] T cce84d138b0542f1a61ca442f9a625f3 P 04e6d9f68e874b24b86fa0750afbed0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:28:35.806195 4084 raft_consensus.cc:3037] T cce84d138b0542f1a61ca442f9a625f3 P 04e6d9f68e874b24b86fa0750afbed0b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:28:35.808460 4084 raft_consensus.cc:504] T cce84d138b0542f1a61ca442f9a625f3 P 04e6d9f68e874b24b86fa0750afbed0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab0adaa90ca48bdb5b49c981a74e557" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43901 } } peers { permanent_uuid: "04e6d9f68e874b24b86fa0750afbed0b" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 36571 } } peers { permanent_uuid: "eea228eefebc4347acb2db74799297a4" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36905 } }
I20260810 23:28:35.809123 4084 leader_election.cc:288] T cce84d138b0542f1a61ca442f9a625f3 P 04e6d9f68e874b24b86fa0750afbed0b [CANDIDATE]: Term 1 election: Requested vote from peers aab0adaa90ca48bdb5b49c981a74e557 (127.14.175.67:43901), eea228eefebc4347acb2db74799297a4 (127.14.175.66:36905)
I20260810 23:28:35.809765 3975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cce84d138b0542f1a61ca442f9a625f3" candidate_uuid: "04e6d9f68e874b24b86fa0750afbed0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aab0adaa90ca48bdb5b49c981a74e557"
I20260810 23:28:35.809765 3751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cce84d138b0542f1a61ca442f9a625f3" candidate_uuid: "04e6d9f68e874b24b86fa0750afbed0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eea228eefebc4347acb2db74799297a4"
I20260810 23:28:35.810009 3975 raft_consensus.cc:3037] T cce84d138b0542f1a61ca442f9a625f3 P aab0adaa90ca48bdb5b49c981a74e557 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:28:35.810009 3751 raft_consensus.cc:3037] T cce84d138b0542f1a61ca442f9a625f3 P eea228eefebc4347acb2db74799297a4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:28:35.813356 3975 raft_consensus.cc:2454] T cce84d138b0542f1a61ca442f9a625f3 P aab0adaa90ca48bdb5b49c981a74e557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04e6d9f68e874b24b86fa0750afbed0b in term 1.
I20260810 23:28:35.813354 3751 raft_consensus.cc:2454] T cce84d138b0542f1a61ca442f9a625f3 P eea228eefebc4347acb2db74799297a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04e6d9f68e874b24b86fa0750afbed0b in term 1.
I20260810 23:28:35.813900 3380 leader_election.cc:302] T cce84d138b0542f1a61ca442f9a625f3 P 04e6d9f68e874b24b86fa0750afbed0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04e6d9f68e874b24b86fa0750afbed0b, eea228eefebc4347acb2db74799297a4; no voters:
I20260810 23:28:35.814294 4084 raft_consensus.cc:2781] T cce84d138b0542f1a61ca442f9a625f3 P 04e6d9f68e874b24b86fa0750afbed0b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:28:35.814805 4084 raft_consensus.cc:686] T cce84d138b0542f1a61ca442f9a625f3 P 04e6d9f68e874b24b86fa0750afbed0b [term 1 LEADER]: Becoming Leader. State: Replica: 04e6d9f68e874b24b86fa0750afbed0b, State: Running, Role: LEADER
I20260810 23:28:35.815340 4084 consensus_queue.cc:237] T cce84d138b0542f1a61ca442f9a625f3 P 04e6d9f68e874b24b86fa0750afbed0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab0adaa90ca48bdb5b49c981a74e557" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43901 } } peers { permanent_uuid: "04e6d9f68e874b24b86fa0750afbed0b" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 36571 } } peers { permanent_uuid: "eea228eefebc4347acb2db74799297a4" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36905 } }
I20260810 23:28:35.820686 3228 catalog_manager.cc:5495] T cce84d138b0542f1a61ca442f9a625f3 P 04e6d9f68e874b24b86fa0750afbed0b reported cstate change: term changed from 0 to 1, leader changed from <none> to 04e6d9f68e874b24b86fa0750afbed0b (127.14.175.65). New cstate: current_term: 1 leader_uuid: "04e6d9f68e874b24b86fa0750afbed0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab0adaa90ca48bdb5b49c981a74e557" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04e6d9f68e874b24b86fa0750afbed0b" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eea228eefebc4347acb2db74799297a4" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36905 } health_report { overall_health: UNKNOWN } } }
I20260810 23:28:35.884445 4068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:35.878983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53692 (local address 127.14.175.65:36571)
0810 23:28:35.879202 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:35.879207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:35.879223 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:28:35.879317 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:35.879321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:35.879400 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:35.879539 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:28:35.879952 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:35.880728 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:35.882207 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:35.882613 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:35.882727 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:35.883283 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:28:35.883799 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:28:35.883958 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:35.884120 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 23:28:35.884208 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 23:28:35.893180 3975 raft_consensus.cc:1261] T cce84d138b0542f1a61ca442f9a625f3 P aab0adaa90ca48bdb5b49c981a74e557 [term 1 FOLLOWER]: Refusing update from remote peer 04e6d9f68e874b24b86fa0750afbed0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:28:35.893185 3751 raft_consensus.cc:1261] T cce84d138b0542f1a61ca442f9a625f3 P eea228eefebc4347acb2db74799297a4 [term 1 FOLLOWER]: Refusing update from remote peer 04e6d9f68e874b24b86fa0750afbed0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:28:35.894340 4091 consensus_queue.cc:1035] T cce84d138b0542f1a61ca442f9a625f3 P 04e6d9f68e874b24b86fa0750afbed0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "eea228eefebc4347acb2db74799297a4" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:28:35.894800 4084 consensus_queue.cc:1035] T cce84d138b0542f1a61ca442f9a625f3 P 04e6d9f68e874b24b86fa0750afbed0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "aab0adaa90ca48bdb5b49c981a74e557" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:28:35.910487 4100 mvcc.cc:205] Tried to move back new op lower bound from 7317112897087299584 to 7317112896783515648. Current Snapshot: MvccSnapshot[applied={T|T < 7317112897087299584}]
I20260810 23:28:35.927675 3229 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:28:35.928153 3229 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:28:35.933807 3229 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff261d817152432bb0581bfb48106ed3: Sending DeleteTablet for 3 replicas of tablet cce84d138b0542f1a61ca442f9a625f3
I20260810 23:28:35.935389 3462 tablet_service.cc:1514] Processing DeleteTablet for tablet cce84d138b0542f1a61ca442f9a625f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:28:35 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:53690
I20260810 23:28:35.935460 3942 tablet_service.cc:1514] Processing DeleteTablet for tablet cce84d138b0542f1a61ca442f9a625f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:28:35 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:44086
I20260810 23:28:35.936462 3718 tablet_service.cc:1514] Processing DeleteTablet for tablet cce84d138b0542f1a61ca442f9a625f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:28:35 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:37812
I20260810 23:28:35.936735 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 3295
I20260810 23:28:35.949688 4112 tablet_replica.cc:330] T cce84d138b0542f1a61ca442f9a625f3 P aab0adaa90ca48bdb5b49c981a74e557: stopping tablet replica
I20260810 23:28:35.951051 4112 raft_consensus.cc:2229] T cce84d138b0542f1a61ca442f9a625f3 P aab0adaa90ca48bdb5b49c981a74e557 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:28:35.952762 4112 raft_consensus.cc:2258] T cce84d138b0542f1a61ca442f9a625f3 P aab0adaa90ca48bdb5b49c981a74e557 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:28:35.954799 4114 tablet_replica.cc:330] T cce84d138b0542f1a61ca442f9a625f3 P eea228eefebc4347acb2db74799297a4: stopping tablet replica
I20260810 23:28:35.955348 4114 raft_consensus.cc:2229] T cce84d138b0542f1a61ca442f9a625f3 P eea228eefebc4347acb2db74799297a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:28:35.956174 4114 raft_consensus.cc:2258] T cce84d138b0542f1a61ca442f9a625f3 P eea228eefebc4347acb2db74799297a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:28:35.958580 4112 ts_tablet_manager.cc:1905] T cce84d138b0542f1a61ca442f9a625f3 P aab0adaa90ca48bdb5b49c981a74e557: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:28:35.958655 4114 ts_tablet_manager.cc:1905] T cce84d138b0542f1a61ca442f9a625f3 P eea228eefebc4347acb2db74799297a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:28:35.966938 4113 tablet_replica.cc:330] T cce84d138b0542f1a61ca442f9a625f3 P 04e6d9f68e874b24b86fa0750afbed0b: stopping tablet replica
I20260810 23:28:35.967646 4113 raft_consensus.cc:2229] T cce84d138b0542f1a61ca442f9a625f3 P 04e6d9f68e874b24b86fa0750afbed0b [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:28:35.968708 4114 ts_tablet_manager.cc:1918] T cce84d138b0542f1a61ca442f9a625f3 P eea228eefebc4347acb2db74799297a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:28:35.968797 4113 raft_consensus.cc:2258] T cce84d138b0542f1a61ca442f9a625f3 P 04e6d9f68e874b24b86fa0750afbed0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:28:35.969029 4114 log.cc:1196] T cce84d138b0542f1a61ca442f9a625f3 P eea228eefebc4347acb2db74799297a4: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-1/wal/wals/cce84d138b0542f1a61ca442f9a625f3
I20260810 23:28:35.970022 4114 ts_tablet_manager.cc:1939] T cce84d138b0542f1a61ca442f9a625f3 P eea228eefebc4347acb2db74799297a4: Deleting consensus metadata
I20260810 23:28:35.971756 4112 ts_tablet_manager.cc:1918] T cce84d138b0542f1a61ca442f9a625f3 P aab0adaa90ca48bdb5b49c981a74e557: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:28:35.972361 4112 log.cc:1196] T cce84d138b0542f1a61ca442f9a625f3 P aab0adaa90ca48bdb5b49c981a74e557: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404299514076-15037-0/minicluster-data/ts-2/wal/wals/cce84d138b0542f1a61ca442f9a625f3
I20260810 23:28:35.972623 3211 catalog_manager.cc:4841] TS eea228eefebc4347acb2db74799297a4 (127.14.175.66:36905): tablet cce84d138b0542f1a61ca442f9a625f3 (table test-table [id=ed4775eff511494db72bb66eca5bc16d]) successfully deleted
I20260810 23:28:35.973397 4112 ts_tablet_manager.cc:1939] T cce84d138b0542f1a61ca442f9a625f3 P aab0adaa90ca48bdb5b49c981a74e557: Deleting consensus metadata
I20260810 23:28:35.973456 4068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:35.938783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53696 (local address 127.14.175.65:36571)
0810 23:28:35.938948 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:35.938953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:35.938972 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:28:35.939030 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:35.939035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:35.939091 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:35.939179 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:35.943935 (+ 4756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:35.946178 (+ 2243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:35.949356 (+ 3178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:35.950528 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:35.950643 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:35.956725 (+ 6082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:35.956742 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:35.956761 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:35.956875 (+ 114us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:35.964047 (+ 7172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:35.965482 (+ 1435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:35.965491 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:35.965495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:35.965923 (+ 428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:35.969760 (+ 3837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:35.969765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:35.969768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:35.970474 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:35.972245 (+ 1771us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:35.972490 (+ 245us) server_negotiation.cc:299] Negotiation successful
0810 23:28:35.972816 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 23:28:35.974545 3573 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:28:35.976724 3209 catalog_manager.cc:4841] TS aab0adaa90ca48bdb5b49c981a74e557 (127.14.175.67:43901): tablet cce84d138b0542f1a61ca442f9a625f3 (table test-table [id=ed4775eff511494db72bb66eca5bc16d]) successfully deleted
I20260810 23:28:35.975776 4113 ts_tablet_manager.cc:1905] T cce84d138b0542f1a61ca442f9a625f3 P 04e6d9f68e874b24b86fa0750afbed0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:28:36.074846 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 3295
W20260810 23:28:36.101842 3209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.175.65:36571 (error 108)
I20260810 23:28:36.102058 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 3588
I20260810 23:28:36.103906 4117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:36.103217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:36571 (local address 127.0.0.1:53708)
0810 23:28:36.103638 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:36.103748 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":216,"threads_started":1}
W20260810 23:28:36.104193 3209 catalog_manager.cc:4584] TS 04e6d9f68e874b24b86fa0750afbed0b (127.14.175.65:36571): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cce84d138b0542f1a61ca442f9a625f3: Network error: Client connection negotiation failed: client connection to 127.14.175.65:36571: connect: Connection refused (error 111)
I20260810 23:28:36.118255 4066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:36.102939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37818 (local address 127.14.175.66:36905)
0810 23:28:36.103112 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:36.103118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:36.103139 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:28:36.103354 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:36.103360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:36.103433 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:36.103522 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:36.105372 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:36.106128 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:36.107882 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:36.108316 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:36.108443 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:36.111833 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:36.111853 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:36.111874 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:36.111922 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:36.115003 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:36.115880 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:36.115889 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:36.115891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:36.116053 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:36.116526 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:36.116530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:36.116532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:36.116956 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:36.117294 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:36.117624 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 23:28:36.117894 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 23:28:36.118948 3814 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:28:36.158180 4117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:36.157696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:36571 (local address 127.0.0.1:53720)
0810 23:28:36.157909 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:36.158035 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260810 23:28:36.159544 4117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:36.159000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:36571 (local address 127.0.0.1:53724)
0810 23:28:36.159167 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:36.159264 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 23:28:36.209842 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 3588
I20260810 23:28:36.220103 4117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:36.219602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:36571 (local address 127.0.0.1:53736)
0810 23:28:36.219830 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:36.219924 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260810 23:28:36.221500 4117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:36.220945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:36571 (local address 127.0.0.1:53742)
0810 23:28:36.221214 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:36.221316 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260810 23:28:36.233539 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 3825
I20260810 23:28:36.246425 4063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:36.234278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44100 (local address 127.14.175.67:43901)
0810 23:28:36.234432 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:36.234437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:36.234459 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:28:36.234621 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:36.234625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:36.234705 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:36.234789 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:36.236171 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:36.236901 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:36.238231 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:36.238640 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:36.238737 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:36.241270 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:36.241286 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:36.241306 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:36.241362 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:36.243948 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:36.244666 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:36.244674 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:36.244676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:36.244818 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:36.245108 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:36.245111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:36.245113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:36.245532 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:36.245772 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:36.245981 (+ 209us) server_negotiation.cc:299] Negotiation successful
0810 23:28:36.246162 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:28:36.247135 4028 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:28:36.302095 4117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:36.301574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:36571 (local address 127.0.0.1:53754)
0810 23:28:36.301809 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:36.301926 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260810 23:28:36.303320 4117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:36.302872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:36571 (local address 127.0.0.1:53766)
0810 23:28:36.303087 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:36.303173 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260810 23:28:36.336751 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 3825
I20260810 23:28:36.360910 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 3179
I20260810 23:28:36.362226 3284 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:28:36.488181 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 3179
I20260810 23:28:36.520653 15037 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 23:28:36.876665 15037 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 23:28:36.876888 15037 mini_ranger.cc:66] Stopping Ranger...
I20260810 23:28:37.232448 15037 mini_ranger.cc:68] Stopped Ranger
2026-08-10 23:28:37.232 UTC [1613] LOG: received smart shutdown request
2026-08-10 23:28:37.235 UTC [1613] LOG: background worker "logical replication launcher" (PID 1629) exited with exit code 1
2026-08-10 23:28:37.236 UTC [1618] LOG: shutting down
2026-08-10 23:28:37.384 UTC [1613] LOG: database system is shut down
2026-08-10T23:28:37Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (55373 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:28:37 dist-test-slave-rmhg krb5kdc[4135](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:28:37 dist-test-slave-rmhg krb5kdc[4135](info): set up 2 sockets
Aug 10 23:28:37 dist-test-slave-rmhg krb5kdc[4135](info): commencing operation
krb5kdc: starting...
W20260810 23:28:38.745371 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.203s user 0.000s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:28:38 dist-test-slave-rmhg krb5kdc[4135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:28:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:28:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
I20260810 23:28:39.020779 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:38599
--webserver_interface=127.14.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:33225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:38599,127.14.175.125:44691,127.14.175.124:34475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:28:39.368597 4159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:28:39.368940 4159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:28:39.369048 4159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:28:39.376507 4159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:28:39.376637 4159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:28:39.376693 4159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:28:39.376749 4159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:28:39.376792 4159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:28:39.387133 4159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:33225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:38599,127.14.175.125:44691,127.14.175.124:34475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:38599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:28:39.388485 4159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:28:39.389837 4159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:28:39.401507 4166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:39.401548 4167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:39.401583 4171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:28:39.402362 4159 server_base.cc:1034] running on GCE node
I20260810 23:28:39.403120 4159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:28:39.404523 4159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:28:39.405727 4159 hybrid_clock.cc:648] HybridClock initialized: now 1786404519405670 us; error 48 us; skew 500 ppm
I20260810 23:28:39.406075 4159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:28:39 dist-test-slave-rmhg krb5kdc[4135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404519, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:28:39.409314 4159 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:28:39.410785 4159 webserver.cc:460] Webserver started at http://127.14.175.126:35017/ using document root <none> and password file <none>
I20260810 23:28:39.411485 4159 fs_manager.cc:359] Metadata directory not provided
I20260810 23:28:39.411607 4159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:28:39.411902 4159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:28:39.415045 4159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "949371114bf047d6b4d94371786da0d2"
format_stamp: "Formatted at 2026-08-10 23:28:39 on dist-test-slave-rmhg"
server_key: "893596e5460740203a3d0d9cf7cd51e2"
server_key_iv: "b46b65da97aca996dbaef18095480f35"
server_key_version: "encryptionkey@0"
I20260810 23:28:39.415876 4159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "949371114bf047d6b4d94371786da0d2"
format_stamp: "Formatted at 2026-08-10 23:28:39 on dist-test-slave-rmhg"
server_key: "893596e5460740203a3d0d9cf7cd51e2"
server_key_iv: "b46b65da97aca996dbaef18095480f35"
server_key_version: "encryptionkey@0"
I20260810 23:28:39.421213 4159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 23:28:39.424717 4179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:39.426090 4159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 23:28:39.426270 4159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "949371114bf047d6b4d94371786da0d2"
format_stamp: "Formatted at 2026-08-10 23:28:39 on dist-test-slave-rmhg"
server_key: "893596e5460740203a3d0d9cf7cd51e2"
server_key_iv: "b46b65da97aca996dbaef18095480f35"
server_key_version: "encryptionkey@0"
I20260810 23:28:39.426457 4159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:28:39.441613 4159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:28:39.446084 4159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:28:39.446370 4159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:28:39.464777 4159 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:38599
I20260810 23:28:39.464824 4264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:38599 every 8 connection(s)
I20260810 23:28:39.466271 4159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:28:39.467286 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 4159
I20260810 23:28:39.467541 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:28:39.468389 15037 external_mini_cluster.cc:1428] Setting key a31fbccf6c2d6a0a101727b6dde77bc8
I20260810 23:28:39.471235 4265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:28:39.480329 4265 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38599 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.175.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.125@KRBTEST.COM" created.
Aug 10 23:28:39 dist-test-slave-rmhg krb5kdc[4135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404519, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:28:39.506963 4273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:39.483817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59424 (local address 127.14.175.126:38599)
0810 23:28:39.485844 (+ 2027us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:39.485853 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:39.485915 (+ 62us) server_negotiation.cc:407] Connection header received
0810 23:28:39.488004 (+ 2089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:39.488010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:39.488305 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:39.488505 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:39.490001 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:39.491278 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:39.492747 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:39.493391 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:39.493592 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:39.499060 (+ 5468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:39.499071 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:39.499087 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:39.499125 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:39.502179 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:39.503001 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:39.503009 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:39.503011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:39.503198 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:39.503618 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:39.503622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:39.503624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:39.504197 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:39.504403 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:39.505010 (+ 607us) server_negotiation.cc:299] Negotiation successful
0810 23:28:39.505417 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":1815,"spinlock_wait_cycles":19200,"thread_start_us":1536,"threads_started":1}
I20260810 23:28:39.507015 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:39.483816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:38599 (local address 127.0.0.1:59424)
0810 23:28:39.484906 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:39.484958 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:39.487565 (+ 2607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:39.488551 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:39.488559 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:39.489384 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:28:39.489786 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:39.489808 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:39.491444 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:39.491448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:39.492594 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:39.492609 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:39.493635 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:39.493643 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:39.493815 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:39.494453 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:28:39.494477 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:28:39.498772 (+ 4295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:28:39.502339 (+ 3567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:39.502354 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:39.502370 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:39.502847 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:39.503314 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:39.503320 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:39.503324 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:39.503502 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:39.504339 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:28:39.504348 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:28:39.504604 (+ 256us) client_negotiation.cc:769] Sending connection context
0810 23:28:39.504852 (+ 248us) client_negotiation.cc:240] Negotiation successful
0810 23:28:39.505541 (+ 689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":178,"thread_start_us":413,"threads_started":1}
I20260810 23:28:39.510480 4265 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.125" port: 44691 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:28:39.511718 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:39.511277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:44691 (local address 127.0.0.1:51168)
0810 23:28:39.511450 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:39.511568 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.125:44691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260810 23:28:39.512012 4180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.175.125:44691: connect: Connection refused (error 111)
I20260810 23:28:39.513675 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:39.513310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:44691 (local address 127.0.0.1:51170)
0810 23:28:39.513452 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:39.513521 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.125:44691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 23:28:39.513880 4265 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.125:44691: Network error: Client connection negotiation failed: client connection to 127.14.175.125:44691: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.125.keytab.
Entry for principal kudu/127.14.175.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.125.keytab.
I20260810 23:28:39.530647 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.125:44691
--webserver_interface=127.14.175.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.125.keytab
--principal=kudu/127.14.175.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:33225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:38599,127.14.175.125:44691,127.14.175.124:34475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:28:39.563315 4265 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.125" port: 44691 } attempt: 1
I20260810 23:28:39.564553 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:39.564067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:44691 (local address 127.0.0.1:51180)
0810 23:28:39.564271 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:39.564400 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.125:44691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260810 23:28:39.565698 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:39.565324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:44691 (local address 127.0.0.1:51190)
0810 23:28:39.565454 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:39.565532 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.125:44691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 23:28:39.565884 4265 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.125:44691: Network error: Client connection negotiation failed: client connection to 127.14.175.125:44691: connect: Connection refused (error 111)
I20260810 23:28:39.634104 4265 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.125" port: 44691 } attempt: 2
I20260810 23:28:39.635406 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:39.634889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:44691 (local address 127.0.0.1:51194)
0810 23:28:39.635146 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:39.635246 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.125:44691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260810 23:28:39.636430 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:39.636085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:44691 (local address 127.0.0.1:51202)
0810 23:28:39.636218 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:39.636289 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.125:44691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 23:28:39.636631 4265 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.125:44691: Network error: Client connection negotiation failed: client connection to 127.14.175.125:44691: connect: Connection refused (error 111)
I20260810 23:28:39.727906 4265 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.125" port: 44691 } attempt: 3
I20260810 23:28:39.729206 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:39.728681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:44691 (local address 127.0.0.1:33150)
0810 23:28:39.728900 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:39.729023 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.125:44691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260810 23:28:39.730268 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:39.729906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:44691 (local address 127.0.0.1:33164)
0810 23:28:39.730058 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:39.730130 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.125:44691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260810 23:28:39.730501 4265 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.125:44691: Network error: Client connection negotiation failed: client connection to 127.14.175.125:44691: connect: Connection refused (error 111)
I20260810 23:28:39.873788 4265 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.125" port: 44691 } attempt: 4
I20260810 23:28:39.875113 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:39.874529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:44691 (local address 127.0.0.1:33176)
0810 23:28:39.874859 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:39.874958 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.125:44691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260810 23:28:39.876349 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:39.875871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:44691 (local address 127.0.0.1:33182)
0810 23:28:39.876053 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:39.876182 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.125:44691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260810 23:28:39.876648 4265 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.125:44691: Network error: Client connection negotiation failed: client connection to 127.14.175.125:44691: connect: Connection refused (error 111)
W20260810 23:28:39.883991 4277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:28:39.884321 4277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:28:39.884424 4277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:28:39.892494 4277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:28:39.892629 4277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:28:39.892683 4277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:28:39.892741 4277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:28:39.892786 4277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:28:39.905244 4277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:33225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:38599,127.14.175.125:44691,127.14.175.124:34475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.125.keytab
--principal=kudu/127.14.175.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.125:44691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.175.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:28:39.908417 4277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:28:39.910300 4277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:28:39.922569 4286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:39.922526 4287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:39.922603 4290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:28:39.923647 4277 server_base.cc:1034] running on GCE node
I20260810 23:28:39.924518 4277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:28:39.926126 4277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:28:39.927364 4277 hybrid_clock.cc:648] HybridClock initialized: now 1786404519927312 us; error 47 us; skew 500 ppm
I20260810 23:28:39.927728 4277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:28:39 dist-test-slave-rmhg krb5kdc[4135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404519, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:28:39.934523 4277 init.cc:377] Logged in from keytab as kudu/127.14.175.125@KRBTEST.COM (short username kudu)
I20260810 23:28:39.936169 4277 webserver.cc:460] Webserver started at http://127.14.175.125:41839/ using document root <none> and password file <none>
I20260810 23:28:39.936878 4277 fs_manager.cc:359] Metadata directory not provided
I20260810 23:28:39.936985 4277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:28:39.937331 4277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:28:39.940475 4277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-1/data/instance:
uuid: "6b4762da897d4b8dbed7cca22d4fffa4"
format_stamp: "Formatted at 2026-08-10 23:28:39 on dist-test-slave-rmhg"
server_key: "4708d49702613fe4394e6800317e4d65"
server_key_iv: "8dca891165e56ce54fbc466323dc3ade"
server_key_version: "encryptionkey@0"
I20260810 23:28:39.941368 4277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-1/wal/instance:
uuid: "6b4762da897d4b8dbed7cca22d4fffa4"
format_stamp: "Formatted at 2026-08-10 23:28:39 on dist-test-slave-rmhg"
server_key: "4708d49702613fe4394e6800317e4d65"
server_key_iv: "8dca891165e56ce54fbc466323dc3ade"
server_key_version: "encryptionkey@0"
I20260810 23:28:39.946908 4277 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 23:28:39.950699 4301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:39.952133 4277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:28:39.952316 4277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-1/wal
uuid: "6b4762da897d4b8dbed7cca22d4fffa4"
format_stamp: "Formatted at 2026-08-10 23:28:39 on dist-test-slave-rmhg"
server_key: "4708d49702613fe4394e6800317e4d65"
server_key_iv: "8dca891165e56ce54fbc466323dc3ade"
server_key_version: "encryptionkey@0"
I20260810 23:28:39.952504 4277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:28:39.982010 4277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:28:39.986967 4277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:28:39.987258 4277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:28:40.006759 4277 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.125:44691
I20260810 23:28:40.006793 4384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.125:44691 every 8 connection(s)
I20260810 23:28:40.008371 4277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-1/data/info.pb
I20260810 23:28:40.008860 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 4277
I20260810 23:28:40.009063 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-1/wal/instance
I20260810 23:28:40.009615 15037 external_mini_cluster.cc:1428] Setting key 6d22febd284b15ce1364422a1b54674f
I20260810 23:28:40.012795 4386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:28:40.020186 4386 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38599 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.175.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.124@KRBTEST.COM" created.
Aug 10 23:28:40 dist-test-slave-rmhg krb5kdc[4135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404519, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.125@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:28:40.046459 4394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.023114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48322 (local address 127.14.175.126:38599)
0810 23:28:40.023659 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:40.023667 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:40.024798 (+ 1131us) server_negotiation.cc:407] Connection header received
0810 23:28:40.027405 (+ 2607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:40.027410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:40.027483 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:40.027611 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:40.029821 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.030699 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.033128 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.033590 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.033712 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:40.040217 (+ 6505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:40.040223 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:40.040226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:40.040280 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:40.043058 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:40.043923 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:40.043928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:40.043930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:40.044063 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:40.044519 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:40.044523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:40.044525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:40.044832 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:40.044973 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:40.045844 (+ 871us) server_negotiation.cc:299] Negotiation successful
0810 23:28:40.046081 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":257,"threads_started":1}
I20260810 23:28:40.047526 4396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.023701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:38599 (local address 127.0.0.1:48322)
0810 23:28:40.024577 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:40.024631 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:40.027094 (+ 2463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:40.027945 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:40.027956 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:40.028697 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:28:40.029601 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:40.029638 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.030864 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.030870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:40.032937 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:40.032954 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.033833 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.033841 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:40.034153 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:40.034887 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:28:40.034918 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:28:40.039919 (+ 5001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:28:40.043235 (+ 3316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:40.043246 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:40.043264 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:40.043780 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:40.044184 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:40.044189 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:40.044193 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:40.044386 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:40.044973 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:28:40.044982 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:28:40.045426 (+ 444us) client_negotiation.cc:769] Sending connection context
0810 23:28:40.045709 (+ 283us) client_negotiation.cc:240] Negotiation successful
0810 23:28:40.046275 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":341,"threads_started":1}
I20260810 23:28:40.049487 4386 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.125" port: 44691 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:28:40 dist-test-slave-rmhg krb5kdc[4135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404519, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.125@KRBTEST.COM for kudu/127.14.175.125@KRBTEST.COM
I20260810 23:28:40.064335 4396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.050293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:44691 (local address 127.0.0.1:33190)
0810 23:28:40.050462 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:40.050481 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:40.050604 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:40.051328 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:40.051332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:40.051819 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:28:40.051958 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:40.051967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.053385 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.053391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:40.054233 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:40.054245 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.055065 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.055071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:40.055278 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:40.055860 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:28:40.055884 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:28:40.058386 (+ 2502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:28:40.061728 (+ 3342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:40.061734 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:40.061738 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:40.062287 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:40.062669 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:40.062674 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:40.062678 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:40.062857 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:40.063460 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:28:40.063465 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:28:40.063637 (+ 172us) client_negotiation.cc:769] Sending connection context
0810 23:28:40.063812 (+ 175us) client_negotiation.cc:240] Negotiation successful
0810 23:28:40.064066 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 23:28:40.064550 4399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.050321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33190 (local address 127.14.175.125:44691)
0810 23:28:40.050913 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:40.050919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:40.050946 (+ 27us) server_negotiation.cc:407] Connection header received
0810 23:28:40.050990 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:40.050997 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:40.051180 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:40.051312 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:40.052125 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.053261 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.054363 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.054922 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.055036 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:40.058608 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:40.058620 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:40.058635 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:40.058666 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:40.061564 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:40.062408 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:40.062416 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:40.062418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:40.062550 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:40.062960 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:40.062963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:40.062965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:40.063326 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:40.063492 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:40.063912 (+ 420us) server_negotiation.cc:299] Negotiation successful
0810 23:28:40.064178 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":447,"thread_start_us":306,"threads_started":1}
I20260810 23:28:40.067334 4386 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.124" port: 34475 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:28:40.068454 4396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.067941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:34475 (local address 127.0.0.1:37718)
0810 23:28:40.068175 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:40.068286 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260810 23:28:40.068717 4304 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.175.124:34475: connect: Connection refused (error 111)
I20260810 23:28:40.070289 4396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.069909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:34475 (local address 127.0.0.1:37722)
0810 23:28:40.070099 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:40.070182 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 23:28:40.070521 4386 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.124:34475: Network error: Client connection negotiation failed: client connection to 127.14.175.124:34475: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.124.keytab.
Entry for principal kudu/127.14.175.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.124.keytab.
I20260810 23:28:40.072587 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.124:34475
--webserver_interface=127.14.175.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.124.keytab
--principal=kudu/127.14.175.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:33225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:38599,127.14.175.125:44691,127.14.175.124:34475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:28:40.119871 4386 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.124" port: 34475 } attempt: 1
I20260810 23:28:40.121263 4396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.120633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:34475 (local address 127.0.0.1:37724)
0810 23:28:40.120871 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:40.121016 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260810 23:28:40.122567 4396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.122132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:34475 (local address 127.0.0.1:37726)
0810 23:28:40.122331 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:40.122428 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 23:28:40.122802 4386 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.124:34475: Network error: Client connection negotiation failed: client connection to 127.14.175.124:34475: connect: Connection refused (error 111)
I20260810 23:28:40.175985 4265 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.125" port: 44691 } attempt: 5
Aug 10 23:28:40 dist-test-slave-rmhg krb5kdc[4135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404519, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for kudu/127.14.175.125@KRBTEST.COM
I20260810 23:28:40.189706 4399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.176874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33198 (local address 127.14.175.125:44691)
0810 23:28:40.177100 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:40.177106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:40.177131 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:28:40.177360 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:40.177364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:40.177450 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:40.177545 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:40.178428 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.179163 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.180385 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.180819 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.180939 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:40.184386 (+ 3447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:40.184392 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:40.184395 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:40.184432 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:40.187073 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:40.187773 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:40.187778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:40.187780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:40.187911 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:40.188294 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:40.188298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:40.188300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:40.188603 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:40.188739 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:40.189140 (+ 401us) server_negotiation.cc:299] Negotiation successful
0810 23:28:40.189386 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260810 23:28:40.189725 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.176783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:44691 (local address 127.0.0.1:33198)
0810 23:28:40.177018 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:40.177043 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:40.177223 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:40.177610 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:40.177615 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:40.178044 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:28:40.178254 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:40.178264 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.179301 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.179305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:40.180235 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:40.180248 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.180992 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.180999 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:40.181217 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:40.181884 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:28:40.181901 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:28:40.184166 (+ 2265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:28:40.187243 (+ 3077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:40.187249 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:40.187254 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:40.187651 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:40.188024 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:40.188030 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:40.188034 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:40.188190 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:40.188726 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:28:40.188730 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:28:40.188886 (+ 156us) client_negotiation.cc:769] Sending connection context
0810 23:28:40.189012 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 23:28:40.189370 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260810 23:28:40.190990 4386 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.124" port: 34475 } attempt: 2
I20260810 23:28:40.191139 4265 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.124" port: 34475 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:28:40.191920 4396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.191486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:34475 (local address 127.0.0.1:37742)
0810 23:28:40.191651 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:40.191747 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 23:28:40.192323 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.191850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:34475 (local address 127.0.0.1:37746)
0810 23:28:40.192041 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:40.192146 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 23:28:40.192876 4396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.192532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:34475 (local address 127.0.0.1:37748)
0810 23:28:40.192670 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:40.192752 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 23:28:40.193122 4386 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.124:34475: Network error: Client connection negotiation failed: client connection to 127.14.175.124:34475: connect: Connection refused (error 111)
I20260810 23:28:40.193754 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.193342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:34475 (local address 127.0.0.1:37760)
0810 23:28:40.193517 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:40.193609 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 23:28:40.193958 4265 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.124:34475: Network error: Client connection negotiation failed: client connection to 127.14.175.124:34475: connect: Connection refused (error 111)
I20260810 23:28:40.245236 4265 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.124" port: 34475 } attempt: 1
I20260810 23:28:40.246676 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.246028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:34475 (local address 127.0.0.1:37764)
0810 23:28:40.246294 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:40.246433 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260810 23:28:40.247857 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.247411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:34475 (local address 127.0.0.1:37780)
0810 23:28:40.247574 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:40.247690 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 23:28:40.248131 4265 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.124:34475: Network error: Client connection negotiation failed: client connection to 127.14.175.124:34475: connect: Connection refused (error 111)
I20260810 23:28:40.284412 4386 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.124" port: 34475 } attempt: 3
I20260810 23:28:40.285625 4396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.285124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:34475 (local address 127.0.0.1:37790)
0810 23:28:40.285348 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:40.285444 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260810 23:28:40.286932 4396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.286466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:34475 (local address 127.0.0.1:37796)
0810 23:28:40.286725 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:40.286808 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
W20260810 23:28:40.287175 4386 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.124:34475: Network error: Client connection negotiation failed: client connection to 127.14.175.124:34475: connect: Connection refused (error 111)
I20260810 23:28:40.316396 4265 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.124" port: 34475 } attempt: 2
I20260810 23:28:40.317690 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.317140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:34475 (local address 127.0.0.1:37802)
0810 23:28:40.317403 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:40.317498 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 23:28:40.318732 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.318409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:34475 (local address 127.0.0.1:37804)
0810 23:28:40.318543 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:40.318621 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 23:28:40.318950 4265 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.124:34475: Network error: Client connection negotiation failed: client connection to 127.14.175.124:34475: connect: Connection refused (error 111)
I20260810 23:28:40.425200 4265 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.124" port: 34475 } attempt: 3
I20260810 23:28:40.426384 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.425909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:34475 (local address 127.0.0.1:37808)
0810 23:28:40.426104 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:40.426200 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260810 23:28:40.427353 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.427038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:34475 (local address 127.0.0.1:37814)
0810 23:28:40.427167 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:40.427249 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 23:28:40.427560 4265 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.124:34475: Network error: Client connection negotiation failed: client connection to 127.14.175.124:34475: connect: Connection refused (error 111)
I20260810 23:28:40.430363 4386 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.124" port: 34475 } attempt: 4
W20260810 23:28:40.430416 4402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:28:40.430760 4402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:28:40.430912 4402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 23:28:40.431406 4396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.430982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:34475 (local address 127.0.0.1:37824)
0810 23:28:40.431182 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:40.431264 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 23:28:40.432265 4396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.431986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:34475 (local address 127.0.0.1:37828)
0810 23:28:40.432093 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:40.432162 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 23:28:40.432448 4386 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.124:34475: Network error: Client connection negotiation failed: client connection to 127.14.175.124:34475: connect: Connection refused (error 111)
W20260810 23:28:40.438791 4402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:28:40.438900 4402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:28:40.438978 4402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:28:40.439057 4402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:28:40.439113 4402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:28:40.449487 4402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:33225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:38599,127.14.175.125:44691,127.14.175.124:34475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.124.keytab
--principal=kudu/127.14.175.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.124:34475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.175.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:28:40.450968 4402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:28:40.452409 4402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:28:40.464811 4409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:40.464988 4413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:40.464880 4411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:28:40.465849 4402 server_base.cc:1034] running on GCE node
I20260810 23:28:40.466830 4402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:28:40.469447 4402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:28:40.470727 4402 hybrid_clock.cc:648] HybridClock initialized: now 1786404520470650 us; error 60 us; skew 500 ppm
I20260810 23:28:40.471184 4402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:28:40 dist-test-slave-rmhg krb5kdc[4135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404520, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:28:40.475127 4402 init.cc:377] Logged in from keytab as kudu/127.14.175.124@KRBTEST.COM (short username kudu)
I20260810 23:28:40.476665 4402 webserver.cc:460] Webserver started at http://127.14.175.124:35067/ using document root <none> and password file <none>
I20260810 23:28:40.477458 4402 fs_manager.cc:359] Metadata directory not provided
I20260810 23:28:40.477567 4402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:28:40.477845 4402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:28:40.480881 4402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-2/data/instance:
uuid: "30b1fa7c3514436da30fde2906cc6fff"
format_stamp: "Formatted at 2026-08-10 23:28:40 on dist-test-slave-rmhg"
server_key: "f53bde85e927c14aa01d86fcd6fec0cb"
server_key_iv: "f8324f426efe8649ba32c7f9803dd1a3"
server_key_version: "encryptionkey@0"
I20260810 23:28:40.481721 4402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-2/wal/instance:
uuid: "30b1fa7c3514436da30fde2906cc6fff"
format_stamp: "Formatted at 2026-08-10 23:28:40 on dist-test-slave-rmhg"
server_key: "f53bde85e927c14aa01d86fcd6fec0cb"
server_key_iv: "f8324f426efe8649ba32c7f9803dd1a3"
server_key_version: "encryptionkey@0"
I20260810 23:28:40.487133 4402 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:28:40.490747 4424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:40.492132 4402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:28:40.492287 4402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-2/wal
uuid: "30b1fa7c3514436da30fde2906cc6fff"
format_stamp: "Formatted at 2026-08-10 23:28:40 on dist-test-slave-rmhg"
server_key: "f53bde85e927c14aa01d86fcd6fec0cb"
server_key_iv: "f8324f426efe8649ba32c7f9803dd1a3"
server_key_version: "encryptionkey@0"
I20260810 23:28:40.492455 4402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:28:40.505366 4402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:28:40.509411 4402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:28:40.509670 4402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:28:40.530387 4402 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.124:34475
I20260810 23:28:40.530417 4516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.124:34475 every 8 connection(s)
I20260810 23:28:40.532074 4402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-2/data/info.pb
I20260810 23:28:40.536497 4517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:28:40.539860 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 4402
I20260810 23:28:40.540028 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404299514076-15037-0/minicluster-data/master-2/wal/instance
I20260810 23:28:40.540535 15037 external_mini_cluster.cc:1428] Setting key df11f4afc30deb608a37acd6fcd4eae1
I20260810 23:28:40.544829 4517 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38599 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:28:40 dist-test-slave-rmhg krb5kdc[4135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.124@KRBTEST.COM
Aug 10 23:28:40 dist-test-slave-rmhg krb5kdc[4135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404520, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.124@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:28:40.564205 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.548916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48336 (local address 127.14.175.126:38599)
0810 23:28:40.549569 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:40.549576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:40.550139 (+ 563us) server_negotiation.cc:407] Connection header received
0810 23:28:40.550333 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:40.550337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:40.550403 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:40.550495 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:40.551753 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.552694 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.554362 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.554971 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.555111 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:40.558566 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:40.558572 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:40.558575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:40.558609 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:40.561062 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:40.561828 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:40.561834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:40.561838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:40.561983 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:40.562390 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:40.562395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:40.562397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:40.562670 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:40.562792 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:40.563545 (+ 753us) server_negotiation.cc:299] Negotiation successful
0810 23:28:40.563777 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":340,"threads_started":1}
I20260810 23:28:40.565460 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.548905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:38599 (local address 127.0.0.1:48336)
0810 23:28:40.549974 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:40.550027 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:40.550208 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:40.550575 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:40.550582 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:40.551324 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:28:40.551582 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:40.551591 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.552857 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.552862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:40.554216 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:40.554228 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.555172 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.555177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:40.555297 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:40.555910 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:28:40.555932 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:28:40.558356 (+ 2424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:28:40.561230 (+ 2874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:40.561242 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:40.561257 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:40.561691 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:40.562094 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:40.562099 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:40.562102 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:40.562272 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:40.562796 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:28:40.562805 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:28:40.563109 (+ 304us) client_negotiation.cc:769] Sending connection context
0810 23:28:40.563418 (+ 309us) client_negotiation.cc:240] Negotiation successful
0810 23:28:40.564038 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":170,"spinlock_wait_cycles":22912,"thread_start_us":597,"threads_started":1}
I20260810 23:28:40.565479 4522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.542631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37834 (local address 127.14.175.124:34475)
0810 23:28:40.543535 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:40.543553 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:40.543611 (+ 58us) server_negotiation.cc:407] Connection header received
0810 23:28:40.545864 (+ 2253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:40.545901 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:40.546551 (+ 650us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:40.547113 (+ 562us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:40.548662 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.550760 (+ 2098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.552385 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.553332 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.553534 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:40.558065 (+ 4531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:40.558077 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:40.558095 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:40.558147 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:40.561103 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:40.561852 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:40.561860 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:40.561862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:40.562015 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:40.562387 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:40.562390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:40.562392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:40.563030 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:40.563327 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:40.563610 (+ 283us) server_negotiation.cc:299] Negotiation successful
0810 23:28:40.563918 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":621,"thread_start_us":336,"threads_started":1}
I20260810 23:28:40.567242 4517 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.125" port: 44691 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:28:40 dist-test-slave-rmhg krb5kdc[4135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404520, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.124@KRBTEST.COM for kudu/127.14.175.125@KRBTEST.COM
Aug 10 23:28:40 dist-test-slave-rmhg krb5kdc[4135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:28:40.581631 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.568013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:44691 (local address 127.0.0.1:33204)
0810 23:28:40.568203 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:40.568223 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:40.568357 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:40.569273 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:40.569277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:40.569631 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:28:40.569798 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:40.569807 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.570798 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.570805 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:40.571710 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:40.571723 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.572450 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.572456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:40.572671 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:40.573272 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:28:40.573291 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:28:40.575557 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:28:40.578975 (+ 3418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:40.578981 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:40.578985 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:40.579421 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:40.579841 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:40.579846 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:40.579848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:40.580008 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:40.580566 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:28:40.580571 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:28:40.580753 (+ 182us) client_negotiation.cc:769] Sending connection context
0810 23:28:40.580892 (+ 139us) client_negotiation.cc:240] Negotiation successful
0810 23:28:40.581261 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 23:28:40.581656 4399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.568130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33204 (local address 127.14.175.125:44691)
0810 23:28:40.568336 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:40.568343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:40.568364 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:28:40.568843 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:40.568847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:40.568929 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:40.569030 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:40.569956 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.570655 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.571857 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.572298 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.572416 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:40.575808 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:40.575813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:40.575817 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:40.575855 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:40.578800 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:40.579563 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:40.579567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:40.579570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:40.579715 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:40.580127 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:40.580130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:40.580132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:40.580436 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:40.580584 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:40.581015 (+ 431us) server_negotiation.cc:299] Negotiation successful
0810 23:28:40.581283 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 23:28:40.582978 4517 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.124" port: 34475 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:28:40.584718 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.569113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48340 (local address 127.14.175.126:38599)
0810 23:28:40.569555 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:40.569560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:40.569572 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:28:40.569658 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:40.569662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:40.569728 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:40.569810 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:40.571088 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.571798 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.573244 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.573675 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.573789 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:40.578462 (+ 4673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:40.578468 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:40.578471 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:40.578521 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:40.581315 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:40.582261 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:40.582267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:40.582271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:40.582454 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:40.582973 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:40.582978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:40.582981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:40.583408 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:40.583694 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:40.584087 (+ 393us) server_negotiation.cc:299] Negotiation successful
0810 23:28:40.584356 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336}
Aug 10 23:28:40 dist-test-slave-rmhg krb5kdc[4135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404520, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.124@KRBTEST.COM for kudu/127.14.175.124@KRBTEST.COM
Aug 10 23:28:40 dist-test-slave-rmhg krb5kdc[4135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.125@KRBTEST.COM
I20260810 23:28:40.598104 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.583802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:34475 (local address 127.0.0.1:37846)
0810 23:28:40.583972 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:40.583991 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:40.584109 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:40.584742 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:40.584745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:40.585216 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:28:40.585414 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:40.585427 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.586570 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.586575 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:40.587757 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:40.587775 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.588535 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.588541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:40.588751 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:40.589431 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:28:40.589448 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:28:40.591693 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:28:40.595375 (+ 3682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:40.595379 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:40.595382 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:40.595826 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:40.596197 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:40.596202 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:40.596206 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:40.596369 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:40.596905 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:28:40.596910 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:28:40.597076 (+ 166us) client_negotiation.cc:769] Sending connection context
0810 23:28:40.597375 (+ 299us) client_negotiation.cc:240] Negotiation successful
0810 23:28:40.597701 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 23:28:40.598448 4522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.584235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37846 (local address 127.14.175.124:34475)
0810 23:28:40.584397 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:40.584403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:40.584419 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:28:40.584481 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:40.584486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:40.584599 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:40.584696 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:40.585591 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.586413 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.587912 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.588358 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.588452 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:40.591913 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:40.591920 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:40.591926 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:40.591962 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:40.595211 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:40.595935 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:40.595939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:40.595941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:40.596073 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:40.596498 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:40.596503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:40.596506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:40.596790 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:40.596958 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:40.597490 (+ 532us) server_negotiation.cc:299] Negotiation successful
0810 23:28:40.597886 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":165,"server-negotiator.queue_time_us":54}
I20260810 23:28:40.603960 4399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.586624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33210 (local address 127.14.175.125:44691)
0810 23:28:40.586800 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:40.586806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:40.586824 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:28:40.587010 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:40.587016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:40.587086 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:40.587159 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:40.588935 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.589878 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.591420 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.591991 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.592125 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:40.597355 (+ 5230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:40.597361 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:40.597365 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:40.597403 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:40.600871 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:40.601737 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:40.601744 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:40.601748 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:40.601969 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:40.602380 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:40.602385 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:40.602389 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:40.602763 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:40.603061 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:40.603323 (+ 262us) server_negotiation.cc:299] Negotiation successful
0810 23:28:40.603629 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 23:28:40.604622 4517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30b1fa7c3514436da30fde2906cc6fff: Bootstrap starting.
I20260810 23:28:40.604818 4265 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.124" port: 34475 } attempt: 4
I20260810 23:28:40.609072 4517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30b1fa7c3514436da30fde2906cc6fff: Neither blocks nor log segments found. Creating new log.
I20260810 23:28:40.609247 15037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:28:40.610456 4517 log.cc:824] T 00000000000000000000000000000000 P 30b1fa7c3514436da30fde2906cc6fff: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:28:40 dist-test-slave-rmhg krb5kdc[4135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404519, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for kudu/127.14.175.124@KRBTEST.COM
I20260810 23:28:40.614050 4517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30b1fa7c3514436da30fde2906cc6fff: No bootstrap required, opened a new log
I20260810 23:28:40.622063 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.605471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:34475 (local address 127.0.0.1:37862)
0810 23:28:40.605749 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:40.605768 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:40.605898 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:40.606264 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:40.606270 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:40.606779 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:28:40.606966 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:40.606978 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.608400 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.608405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:40.609662 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:40.609677 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.610673 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.610679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:40.610874 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:40.611621 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:28:40.611648 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:28:40.614514 (+ 2866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:28:40.618792 (+ 4278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:40.618800 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:40.618804 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:40.619378 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:40.619858 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:40.619864 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:40.619868 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:40.620068 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:40.620861 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:28:40.620867 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:28:40.621088 (+ 221us) client_negotiation.cc:769] Sending connection context
0810 23:28:40.621292 (+ 204us) client_negotiation.cc:240] Negotiation successful
0810 23:28:40.621616 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260810 23:28:40.623338 4522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.605511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37862 (local address 127.14.175.124:34475)
0810 23:28:40.605678 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:40.605684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:40.605835 (+ 151us) server_negotiation.cc:407] Connection header received
0810 23:28:40.606019 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:40.606025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:40.606107 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:40.606210 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:40.607121 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.608186 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.609809 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.610516 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.610646 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:40.614736 (+ 4090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:40.614744 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:40.614749 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:40.614791 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:40.618599 (+ 3808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:40.619536 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:40.619543 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:40.619547 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:40.619720 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:40.620199 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:40.620204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:40.620208 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:40.620644 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:40.620794 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:40.622696 (+ 1902us) server_negotiation.cc:299] Negotiation successful
0810 23:28:40.622956 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:28:40.629583 4265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2: Bootstrap starting.
I20260810 23:28:40.634269 4265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2: Neither blocks nor log segments found. Creating new log.
I20260810 23:28:40.635769 4265 log.cc:824] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2: Log is configured to *not* fsync() on all Append() calls
I20260810 23:28:40.639780 4399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.615203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33214 (local address 127.14.175.125:44691)
0810 23:28:40.615375 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:40.615381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:40.616482 (+ 1101us) server_negotiation.cc:407] Connection header received
0810 23:28:40.616766 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:40.616773 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:40.616859 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:40.616926 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:40.619381 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.620526 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.624614 (+ 4088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.625409 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.625518 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:40.632661 (+ 7143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:40.632669 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:40.632674 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:40.632720 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:40.636367 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:40.637291 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:40.637297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:40.637301 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:40.637469 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:40.637945 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:40.637951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:40.637954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:40.638314 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:40.638546 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:40.639167 (+ 621us) server_negotiation.cc:299] Negotiation successful
0810 23:28:40.639409 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 23:28:40.639488 4517 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30b1fa7c3514436da30fde2906cc6fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949371114bf047d6b4d94371786da0d2" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38599 } } peers { permanent_uuid: "6b4762da897d4b8dbed7cca22d4fffa4" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 44691 } } peers { permanent_uuid: "30b1fa7c3514436da30fde2906cc6fff" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 34475 } }
I20260810 23:28:40.639933 4517 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30b1fa7c3514436da30fde2906cc6fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:28:40.641351 4517 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30b1fa7c3514436da30fde2906cc6fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30b1fa7c3514436da30fde2906cc6fff, State: Initialized, Role: FOLLOWER
I20260810 23:28:40.642338 4517 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30b1fa7c3514436da30fde2906cc6fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949371114bf047d6b4d94371786da0d2" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38599 } } peers { permanent_uuid: "6b4762da897d4b8dbed7cca22d4fffa4" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 44691 } } peers { permanent_uuid: "30b1fa7c3514436da30fde2906cc6fff" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 34475 } }
I20260810 23:28:40.643980 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.614640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48350 (local address 127.14.175.126:38599)
0810 23:28:40.614827 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:40.614833 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:40.623457 (+ 8624us) server_negotiation.cc:407] Connection header received
0810 23:28:40.626676 (+ 3219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:40.626683 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:40.626771 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:40.626851 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:40.629620 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.630748 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.632938 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.633624 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.633723 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:40.636941 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:40.636949 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:40.636953 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:40.636996 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:40.640507 (+ 3511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:40.641322 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:40.641328 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:40.641331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:40.641483 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:40.642010 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:40.642016 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:40.642019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:40.642396 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:40.642660 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:40.643313 (+ 653us) server_negotiation.cc:299] Negotiation successful
0810 23:28:40.643570 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 23:28:40.647578 4265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2: No bootstrap required, opened a new log
I20260810 23:28:40.652482 4522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.616524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37874 (local address 127.14.175.124:34475)
0810 23:28:40.623573 (+ 7049us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:40.623578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:40.624256 (+ 678us) server_negotiation.cc:407] Connection header received
0810 23:28:40.624425 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:40.624430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:40.624551 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:40.624643 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:40.631539 (+ 6896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.632576 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.640476 (+ 7900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.641220 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.641361 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:40.644747 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:40.644756 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:40.644761 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:40.644801 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:40.648315 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:40.649377 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:40.649382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:40.649386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:40.649542 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:40.650016 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:40.650021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:40.650024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:40.650384 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:40.650655 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:40.651847 (+ 1192us) server_negotiation.cc:299] Negotiation successful
0810 23:28:40.652122 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6954,"thread_start_us":12712,"threads_started":1}
I20260810 23:28:40.653582 4541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30b1fa7c3514436da30fde2906cc6fff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949371114bf047d6b4d94371786da0d2" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38599 } } peers { permanent_uuid: "6b4762da897d4b8dbed7cca22d4fffa4" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 44691 } } peers { permanent_uuid: "30b1fa7c3514436da30fde2906cc6fff" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 34475 } } }
I20260810 23:28:40.654248 4541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30b1fa7c3514436da30fde2906cc6fff [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:28:40.655253 4517 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30b1fa7c3514436da30fde2906cc6fff [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:28:40.657764 4265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949371114bf047d6b4d94371786da0d2" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38599 } } peers { permanent_uuid: "6b4762da897d4b8dbed7cca22d4fffa4" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 44691 } } peers { permanent_uuid: "30b1fa7c3514436da30fde2906cc6fff" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 34475 } }
I20260810 23:28:40.658102 4265 raft_consensus.cc:374] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:28:40.658264 4265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 949371114bf047d6b4d94371786da0d2, State: Initialized, Role: FOLLOWER
I20260810 23:28:40.659060 4265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949371114bf047d6b4d94371786da0d2" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38599 } } peers { permanent_uuid: "6b4762da897d4b8dbed7cca22d4fffa4" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 44691 } } peers { permanent_uuid: "30b1fa7c3514436da30fde2906cc6fff" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 34475 } }
I20260810 23:28:40.662426 4552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949371114bf047d6b4d94371786da0d2" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38599 } } peers { permanent_uuid: "6b4762da897d4b8dbed7cca22d4fffa4" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 44691 } } peers { permanent_uuid: "30b1fa7c3514436da30fde2906cc6fff" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 34475 } } }
I20260810 23:28:40.662909 4552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:28:40.663789 4265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 23:28:40.673934 4560 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 30b1fa7c3514436da30fde2906cc6fff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:28:40.674252 4560 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:28:40.668936 4402 master_runner.cc:186] Error getting master registration for 127.14.175.126:38599: OK, instance_id { permanent_uuid: "949371114bf047d6b4d94371786da0d2" instance_seqno: 1786404519459686 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:28:40.680231 4402 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 23:28:40.683950 4577 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:28:40.685078 4577 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:28:40.682714 4159 master_runner.cc:186] Error getting master registration for 127.14.175.125:44691: OK, instance_id { permanent_uuid: "6b4762da897d4b8dbed7cca22d4fffa4" instance_seqno: 1786404520001107 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:28:40.691608 4159 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 23:28:40.699303 4187 tablet.cc:2369] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:28:40.731667 4386 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.124" port: 34475 } attempt: 5
Aug 10 23:28:40 dist-test-slave-rmhg krb5kdc[4135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404519, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.125@KRBTEST.COM for kudu/127.14.175.124@KRBTEST.COM
I20260810 23:28:40.745810 4396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.732328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:34475 (local address 127.0.0.1:37878)
0810 23:28:40.732592 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:40.732613 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:40.732733 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:40.733072 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:40.733077 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:40.733559 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:28:40.733729 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:40.733739 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.734819 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.734826 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:40.735725 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:40.735738 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.736478 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.736484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:40.736712 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:40.737331 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:28:40.737350 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:28:40.739435 (+ 2085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:28:40.742785 (+ 3350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:40.742791 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:40.742796 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:40.743180 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:40.743723 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:40.743728 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:40.743731 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:40.743920 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:40.744703 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:28:40.744707 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:28:40.744917 (+ 210us) client_negotiation.cc:769] Sending connection context
0810 23:28:40.745068 (+ 151us) client_negotiation.cc:240] Negotiation successful
0810 23:28:40.745432 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260810 23:28:40.746781 4522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:40.732412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37878 (local address 127.14.175.124:34475)
0810 23:28:40.732595 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:40.732601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:40.732670 (+ 69us) server_negotiation.cc:407] Connection header received
0810 23:28:40.732858 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:40.732862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:40.732925 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:40.733010 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:40.733890 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.734666 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.735879 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:40.736335 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:40.736452 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:40.739641 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:40.739648 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:40.739652 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:40.739690 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:40.742647 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:40.743362 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:40.743367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:40.743370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:40.743559 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:40.744050 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:40.744054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:40.744057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:40.744432 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:40.744622 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:40.746118 (+ 1496us) server_negotiation.cc:299] Negotiation successful
0810 23:28:40.746376 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 23:28:40.751896 4386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b4762da897d4b8dbed7cca22d4fffa4: Bootstrap starting.
I20260810 23:28:40.755569 4386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b4762da897d4b8dbed7cca22d4fffa4: Neither blocks nor log segments found. Creating new log.
I20260810 23:28:40.756925 4386 log.cc:824] T 00000000000000000000000000000000 P 6b4762da897d4b8dbed7cca22d4fffa4: Log is configured to *not* fsync() on all Append() calls
I20260810 23:28:40.760064 4386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b4762da897d4b8dbed7cca22d4fffa4: No bootstrap required, opened a new log
W20260810 23:28:40.761823 4438 tablet.cc:2369] T 00000000000000000000000000000000 P 30b1fa7c3514436da30fde2906cc6fff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:28:40.770846 4386 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b4762da897d4b8dbed7cca22d4fffa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949371114bf047d6b4d94371786da0d2" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38599 } } peers { permanent_uuid: "6b4762da897d4b8dbed7cca22d4fffa4" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 44691 } } peers { permanent_uuid: "30b1fa7c3514436da30fde2906cc6fff" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 34475 } }
I20260810 23:28:40.771232 4386 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b4762da897d4b8dbed7cca22d4fffa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:28:40.771354 4386 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b4762da897d4b8dbed7cca22d4fffa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b4762da897d4b8dbed7cca22d4fffa4, State: Initialized, Role: FOLLOWER
I20260810 23:28:40.772043 4386 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b4762da897d4b8dbed7cca22d4fffa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949371114bf047d6b4d94371786da0d2" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38599 } } peers { permanent_uuid: "6b4762da897d4b8dbed7cca22d4fffa4" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 44691 } } peers { permanent_uuid: "30b1fa7c3514436da30fde2906cc6fff" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 34475 } }
I20260810 23:28:40.773414 4552 raft_consensus.cc:482] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:28:40.773780 4552 raft_consensus.cc:504] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949371114bf047d6b4d94371786da0d2" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38599 } } peers { permanent_uuid: "6b4762da897d4b8dbed7cca22d4fffa4" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 44691 } } peers { permanent_uuid: "30b1fa7c3514436da30fde2906cc6fff" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 34475 } }
I20260810 23:28:40.774576 4386 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b4762da897d4b8dbed7cca22d4fffa4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:28:40.775601 4552 leader_election.cc:288] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b4762da897d4b8dbed7cca22d4fffa4 (127.14.175.125:44691), 30b1fa7c3514436da30fde2906cc6fff (127.14.175.124:34475)
I20260810 23:28:40.776026 4582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b4762da897d4b8dbed7cca22d4fffa4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949371114bf047d6b4d94371786da0d2" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38599 } } peers { permanent_uuid: "6b4762da897d4b8dbed7cca22d4fffa4" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 44691 } } peers { permanent_uuid: "30b1fa7c3514436da30fde2906cc6fff" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 34475 } } }
I20260810 23:28:40.776562 4582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b4762da897d4b8dbed7cca22d4fffa4 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:28:40.776854 4351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "949371114bf047d6b4d94371786da0d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b4762da897d4b8dbed7cca22d4fffa4" is_pre_election: true
I20260810 23:28:40.778434 4476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "949371114bf047d6b4d94371786da0d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30b1fa7c3514436da30fde2906cc6fff" is_pre_election: true
I20260810 23:28:40.779063 4476 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 30b1fa7c3514436da30fde2906cc6fff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 949371114bf047d6b4d94371786da0d2 in term 0.
I20260810 23:28:40.781657 4180 leader_election.cc:302] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30b1fa7c3514436da30fde2906cc6fff, 949371114bf047d6b4d94371786da0d2; no voters:
I20260810 23:28:40.782090 4552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:28:40.782276 4552 raft_consensus.cc:482] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:28:40.782387 4552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:28:40.782394 4351 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6b4762da897d4b8dbed7cca22d4fffa4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 949371114bf047d6b4d94371786da0d2 in term 0.
I20260810 23:28:40.787642 4552 raft_consensus.cc:504] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949371114bf047d6b4d94371786da0d2" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38599 } } peers { permanent_uuid: "6b4762da897d4b8dbed7cca22d4fffa4" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 44691 } } peers { permanent_uuid: "30b1fa7c3514436da30fde2906cc6fff" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 34475 } }
I20260810 23:28:40.789551 4351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "949371114bf047d6b4d94371786da0d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b4762da897d4b8dbed7cca22d4fffa4"
I20260810 23:28:40.789892 4351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b4762da897d4b8dbed7cca22d4fffa4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:28:40.793861 4351 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6b4762da897d4b8dbed7cca22d4fffa4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 949371114bf047d6b4d94371786da0d2 in term 1.
I20260810 23:28:40.794596 4180 leader_election.cc:302] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b4762da897d4b8dbed7cca22d4fffa4, 949371114bf047d6b4d94371786da0d2; no voters:
I20260810 23:28:40.799734 4552 leader_election.cc:288] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2 [CANDIDATE]: Term 1 election: Requested vote from peers 6b4762da897d4b8dbed7cca22d4fffa4 (127.14.175.125:44691), 30b1fa7c3514436da30fde2906cc6fff (127.14.175.124:34475)
I20260810 23:28:40.800417 4552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:28:40.800953 4552 raft_consensus.cc:686] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2 [term 1 LEADER]: Becoming Leader. State: Replica: 949371114bf047d6b4d94371786da0d2, State: Running, Role: LEADER
I20260810 23:28:40.801891 4552 consensus_queue.cc:237] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949371114bf047d6b4d94371786da0d2" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38599 } } peers { permanent_uuid: "6b4762da897d4b8dbed7cca22d4fffa4" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 44691 } } peers { permanent_uuid: "30b1fa7c3514436da30fde2906cc6fff" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 34475 } }
I20260810 23:28:40.812155 4476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "949371114bf047d6b4d94371786da0d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30b1fa7c3514436da30fde2906cc6fff"
I20260810 23:28:40.812489 4476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30b1fa7c3514436da30fde2906cc6fff [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:28:40.816411 4476 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 30b1fa7c3514436da30fde2906cc6fff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 949371114bf047d6b4d94371786da0d2 in term 1.
I20260810 23:28:40.822813 4585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 949371114bf047d6b4d94371786da0d2. Latest consensus state: current_term: 1 leader_uuid: "949371114bf047d6b4d94371786da0d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949371114bf047d6b4d94371786da0d2" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38599 } } peers { permanent_uuid: "6b4762da897d4b8dbed7cca22d4fffa4" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 44691 } } peers { permanent_uuid: "30b1fa7c3514436da30fde2906cc6fff" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 34475 } } }
I20260810 23:28:40.823454 4585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2 [sys.catalog]: This master's current role is: LEADER
I20260810 23:28:40.830864 4591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:28:40.832811 4591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:28:40.864334 4476 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 30b1fa7c3514436da30fde2906cc6fff [term 1 FOLLOWER]: Refusing update from remote peer 949371114bf047d6b4d94371786da0d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:28:40.865432 4585 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30b1fa7c3514436da30fde2906cc6fff" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 34475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:28:40.877696 4351 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6b4762da897d4b8dbed7cca22d4fffa4 [term 1 FOLLOWER]: Refusing update from remote peer 949371114bf047d6b4d94371786da0d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:28:40.878867 4585 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b4762da897d4b8dbed7cca22d4fffa4" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:28:40.910492 4582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b4762da897d4b8dbed7cca22d4fffa4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 949371114bf047d6b4d94371786da0d2. Latest consensus state: current_term: 1 leader_uuid: "949371114bf047d6b4d94371786da0d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949371114bf047d6b4d94371786da0d2" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38599 } } peers { permanent_uuid: "6b4762da897d4b8dbed7cca22d4fffa4" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 44691 } } peers { permanent_uuid: "30b1fa7c3514436da30fde2906cc6fff" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 34475 } } }
I20260810 23:28:40.910877 4582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b4762da897d4b8dbed7cca22d4fffa4 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:28:40.921645 4585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "949371114bf047d6b4d94371786da0d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949371114bf047d6b4d94371786da0d2" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38599 } } peers { permanent_uuid: "6b4762da897d4b8dbed7cca22d4fffa4" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 44691 } } peers { permanent_uuid: "30b1fa7c3514436da30fde2906cc6fff" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 34475 } } }
I20260810 23:28:40.921967 4585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2 [sys.catalog]: This master's current role is: LEADER
I20260810 23:28:40.922322 4595 mvcc.cc:205] Tried to move back new op lower bound from 7317112917426016256 to 7317112917219217408. Current Snapshot: MvccSnapshot[applied={T|T < 7317112917426016256}]
I20260810 23:28:40.927878 4591 catalog_manager.cc:1347] Generated new cluster ID: 1d96b4d268384e9a94edc50584c8a9ab
I20260810 23:28:40.927760 4582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b4762da897d4b8dbed7cca22d4fffa4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "949371114bf047d6b4d94371786da0d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949371114bf047d6b4d94371786da0d2" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38599 } } peers { permanent_uuid: "6b4762da897d4b8dbed7cca22d4fffa4" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 44691 } } peers { permanent_uuid: "30b1fa7c3514436da30fde2906cc6fff" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 34475 } } }
I20260810 23:28:40.928007 4591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:28:40.928081 4582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b4762da897d4b8dbed7cca22d4fffa4 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:28:40.928107 4608 mvcc.cc:205] Tried to move back new op lower bound from 7317112917426016256 to 7317112917219217408. Current Snapshot: MvccSnapshot[applied={T|T < 7317112917426016256}]
I20260810 23:28:40.932502 4541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30b1fa7c3514436da30fde2906cc6fff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 949371114bf047d6b4d94371786da0d2. Latest consensus state: current_term: 1 leader_uuid: "949371114bf047d6b4d94371786da0d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949371114bf047d6b4d94371786da0d2" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38599 } } peers { permanent_uuid: "6b4762da897d4b8dbed7cca22d4fffa4" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 44691 } } peers { permanent_uuid: "30b1fa7c3514436da30fde2906cc6fff" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 34475 } } }
I20260810 23:28:40.932848 4541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30b1fa7c3514436da30fde2906cc6fff [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:28:40.935711 4552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "949371114bf047d6b4d94371786da0d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949371114bf047d6b4d94371786da0d2" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38599 } } peers { permanent_uuid: "6b4762da897d4b8dbed7cca22d4fffa4" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 44691 } } peers { permanent_uuid: "30b1fa7c3514436da30fde2906cc6fff" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 34475 } } }
I20260810 23:28:40.936070 4552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2 [sys.catalog]: This master's current role is: LEADER
I20260810 23:28:40.943418 4607 mvcc.cc:205] Tried to move back new op lower bound from 7317112917426016256 to 7317112917219217408. Current Snapshot: MvccSnapshot[applied={T|T < 7317112917426016256}]
I20260810 23:28:40.945755 4541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30b1fa7c3514436da30fde2906cc6fff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "949371114bf047d6b4d94371786da0d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "949371114bf047d6b4d94371786da0d2" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38599 } } peers { permanent_uuid: "6b4762da897d4b8dbed7cca22d4fffa4" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 44691 } } peers { permanent_uuid: "30b1fa7c3514436da30fde2906cc6fff" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 34475 } } }
I20260810 23:28:40.946044 4541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30b1fa7c3514436da30fde2906cc6fff [sys.catalog]: This master's current role is: FOLLOWER
W20260810 23:28:40.956561 4602 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6b4762da897d4b8dbed7cca22d4fffa4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:28:40.956822 4602 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:28:40.959784 4591 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:28:40.961711 4591 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:28:40.978328 4591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 949371114bf047d6b4d94371786da0d2: Generated new TSK 0
I20260810 23:28:40.979673 4591 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:28:41.676501 4560 catalog_manager.cc:1259] Loaded cluster ID: 1d96b4d268384e9a94edc50584c8a9ab
I20260810 23:28:41.676609 4560 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 30b1fa7c3514436da30fde2906cc6fff: loading cluster ID for follower catalog manager: success
I20260810 23:28:41.679171 4560 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 30b1fa7c3514436da30fde2906cc6fff: acquiring CA information for follower catalog manager: success
I20260810 23:28:41.681063 4560 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 30b1fa7c3514436da30fde2906cc6fff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:28:41.958791 4602 catalog_manager.cc:1259] Loaded cluster ID: 1d96b4d268384e9a94edc50584c8a9ab
I20260810 23:28:41.958896 4602 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6b4762da897d4b8dbed7cca22d4fffa4: loading cluster ID for follower catalog manager: success
I20260810 23:28:41.961480 4602 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6b4762da897d4b8dbed7cca22d4fffa4: acquiring CA information for follower catalog manager: success
I20260810 23:28:41.963335 4602 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6b4762da897d4b8dbed7cca22d4fffa4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:28:42.132910 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 4159
I20260810 23:28:42.134536 4258 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:28:42.246905 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 4159
I20260810 23:28:42.277737 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 4277
I20260810 23:28:42.278884 4379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:28:42.387786 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 4277
I20260810 23:28:42.415812 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 4402
I20260810 23:28:42.416954 4510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:28:42.524789 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 4402
2026-08-10T23:28:42Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5028 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:28:42 dist-test-slave-rmhg krb5kdc[4630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:28:42 dist-test-slave-rmhg krb5kdc[4630](info): set up 2 sockets
Aug 10 23:28:42 dist-test-slave-rmhg krb5kdc[4630](info): commencing operation
krb5kdc: starting...
W20260810 23:28:43.758843 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.188s user 0.004s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:28:43 dist-test-slave-rmhg krb5kdc[4630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:28:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:28:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
I20260810 23:28:44.037744 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:43671
--webserver_interface=127.14.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:42471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:43671
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:28:44.406666 4647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:28:44.406970 4647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:28:44.407104 4647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:28:44.414376 4647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:28:44.414500 4647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:28:44.414553 4647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:28:44.414633 4647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:28:44.414691 4647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:28:44.424767 4647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:42471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:43671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:43671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:28:44.426211 4647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:28:44.427632 4647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:28:44.440429 4653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:44.440490 4655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:44.440507 4652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:28:44.441124 4647 server_base.cc:1034] running on GCE node
I20260810 23:28:44.442111 4647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:28:44.443723 4647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:28:44.444972 4647 hybrid_clock.cc:648] HybridClock initialized: now 1786404524444921 us; error 34 us; skew 500 ppm
I20260810 23:28:44.445406 4647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:28:44 dist-test-slave-rmhg krb5kdc[4630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404524, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:28:44.448712 4647 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:28:44.450435 4647 webserver.cc:460] Webserver started at http://127.14.175.126:38581/ using document root <none> and password file <none>
I20260810 23:28:44.451177 4647 fs_manager.cc:359] Metadata directory not provided
I20260810 23:28:44.451294 4647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:28:44.451639 4647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:28:44.454872 4647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "64925b81f63c45cdb4d5685293dfb37c"
format_stamp: "Formatted at 2026-08-10 23:28:44 on dist-test-slave-rmhg"
server_key: "6f6800cc92fe1dd6f605c99710bd24c5"
server_key_iv: "9c455fb4d02c4e569db03a855df45f43"
server_key_version: "encryptionkey@0"
I20260810 23:28:44.455701 4647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "64925b81f63c45cdb4d5685293dfb37c"
format_stamp: "Formatted at 2026-08-10 23:28:44 on dist-test-slave-rmhg"
server_key: "6f6800cc92fe1dd6f605c99710bd24c5"
server_key_iv: "9c455fb4d02c4e569db03a855df45f43"
server_key_version: "encryptionkey@0"
I20260810 23:28:44.461336 4647 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260810 23:28:44.464955 4672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:44.466387 4647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:28:44.466555 4647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "64925b81f63c45cdb4d5685293dfb37c"
format_stamp: "Formatted at 2026-08-10 23:28:44 on dist-test-slave-rmhg"
server_key: "6f6800cc92fe1dd6f605c99710bd24c5"
server_key_iv: "9c455fb4d02c4e569db03a855df45f43"
server_key_version: "encryptionkey@0"
I20260810 23:28:44.466766 4647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:28:44.490419 4647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:28:44.494817 4647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:28:44.495119 4647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:28:44.514709 4647 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:43671
I20260810 23:28:44.514729 4760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:43671 every 8 connection(s)
I20260810 23:28:44.516211 4647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:28:44.517884 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 4647
I20260810 23:28:44.518126 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:28:44.519039 15037 external_mini_cluster.cc:1428] Setting key 45422ae6b8d437fcdc2fe3bd3a970eef
I20260810 23:28:44.521341 4762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 23:28:44 dist-test-slave-rmhg krb5kdc[4630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:28:44.534816 4762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64925b81f63c45cdb4d5685293dfb37c: Bootstrap starting.
I20260810 23:28:44.538967 4762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64925b81f63c45cdb4d5685293dfb37c: Neither blocks nor log segments found. Creating new log.
I20260810 23:28:44.540283 4762 log.cc:824] T 00000000000000000000000000000000 P 64925b81f63c45cdb4d5685293dfb37c: Log is configured to *not* fsync() on all Append() calls
I20260810 23:28:44.543839 4762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64925b81f63c45cdb4d5685293dfb37c: No bootstrap required, opened a new log
I20260810 23:28:44.544468 4766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:44.521040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38190 (local address 127.14.175.126:43671)
0810 23:28:44.521875 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:44.521888 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:44.521930 (+ 42us) server_negotiation.cc:407] Connection header received
0810 23:28:44.523655 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:44.523679 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:44.524082 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:44.524443 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:44.526427 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:44.527874 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:44.529708 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:44.530431 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:44.530571 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:44.535666 (+ 5095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:44.535679 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:44.535700 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:44.535748 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:44.539647 (+ 3899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:44.540444 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:44.540453 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:44.540462 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:44.540691 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:44.541130 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:44.541135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:44.541138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:44.541882 (+ 744us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:44.542183 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:44.542619 (+ 436us) server_negotiation.cc:299] Negotiation successful
0810 23:28:44.543074 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":361,"threads_started":1}
I20260810 23:28:44.548372 15037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:28:44.553221 4762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64925b81f63c45cdb4d5685293dfb37c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64925b81f63c45cdb4d5685293dfb37c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43671 } }
I20260810 23:28:44.553642 4762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64925b81f63c45cdb4d5685293dfb37c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:28:44.553725 4762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64925b81f63c45cdb4d5685293dfb37c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64925b81f63c45cdb4d5685293dfb37c, State: Initialized, Role: FOLLOWER
I20260810 23:28:44.554283 4762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64925b81f63c45cdb4d5685293dfb37c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64925b81f63c45cdb4d5685293dfb37c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43671 } }
I20260810 23:28:44.554491 4762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64925b81f63c45cdb4d5685293dfb37c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:28:44.554590 4762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64925b81f63c45cdb4d5685293dfb37c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:28:44.554714 4762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64925b81f63c45cdb4d5685293dfb37c [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:28:44.556898 4762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64925b81f63c45cdb4d5685293dfb37c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64925b81f63c45cdb4d5685293dfb37c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43671 } }
I20260810 23:28:44.557420 4762 leader_election.cc:302] T 00000000000000000000000000000000 P 64925b81f63c45cdb4d5685293dfb37c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64925b81f63c45cdb4d5685293dfb37c; no voters:
I20260810 23:28:44.557962 4762 leader_election.cc:288] T 00000000000000000000000000000000 P 64925b81f63c45cdb4d5685293dfb37c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:28:44.558137 4770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64925b81f63c45cdb4d5685293dfb37c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:28:44.558815 4770 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64925b81f63c45cdb4d5685293dfb37c [term 1 LEADER]: Becoming Leader. State: Replica: 64925b81f63c45cdb4d5685293dfb37c, State: Running, Role: LEADER
I20260810 23:28:44.559379 4770 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64925b81f63c45cdb4d5685293dfb37c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64925b81f63c45cdb4d5685293dfb37c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43671 } }
I20260810 23:28:44.560011 4762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64925b81f63c45cdb4d5685293dfb37c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:28:44.562330 4772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64925b81f63c45cdb4d5685293dfb37c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64925b81f63c45cdb4d5685293dfb37c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64925b81f63c45cdb4d5685293dfb37c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43671 } } }
I20260810 23:28:44.562300 4773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64925b81f63c45cdb4d5685293dfb37c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64925b81f63c45cdb4d5685293dfb37c. Latest consensus state: current_term: 1 leader_uuid: "64925b81f63c45cdb4d5685293dfb37c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64925b81f63c45cdb4d5685293dfb37c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43671 } } }
I20260810 23:28:44.562789 4772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64925b81f63c45cdb4d5685293dfb37c [sys.catalog]: This master's current role is: LEADER
I20260810 23:28:44.562815 4773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64925b81f63c45cdb4d5685293dfb37c [sys.catalog]: This master's current role is: LEADER
I20260810 23:28:44.569324 4780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:28:44.573951 4780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:28:44.582455 4780 catalog_manager.cc:1347] Generated new cluster ID: a875dbb261364271b62ac41f33353f52
I20260810 23:28:44.582589 4780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:28:44.607220 4780 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:28:44.608664 4780 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:28:44.618659 4780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64925b81f63c45cdb4d5685293dfb37c: Generated new TSK 0
I20260810 23:28:44.619597 4780 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:28:44.661489 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 4647
I20260810 23:28:44.662859 4755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:28:44.761775 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 4647
2026-08-10T23:28:44Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2234 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:28:44 dist-test-slave-rmhg krb5kdc[4807](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:28:44 dist-test-slave-rmhg krb5kdc[4807](info): set up 2 sockets
Aug 10 23:28:44 dist-test-slave-rmhg krb5kdc[4807](info): commencing operation
krb5kdc: starting...
W20260810 23:28:45.996737 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.192s user 0.008s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:28:46 dist-test-slave-rmhg krb5kdc[4807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:28:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:28:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
I20260810 23:28:46.285270 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:42557
--webserver_interface=127.14.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:42173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:42557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:28:46.634676 4832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:28:46.634981 4832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:28:46.635071 4832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:28:46.642485 4832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:28:46.642613 4832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:28:46.642670 4832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:28:46.642727 4832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:28:46.642771 4832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:28:46.652998 4832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:42173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:42557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:42557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:28:46.654388 4832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:28:46.655782 4832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:28:46.667930 4845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:46.667870 4848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:46.668452 4851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:28:46.669868 4832 server_base.cc:1034] running on GCE node
I20260810 23:28:46.670725 4832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:28:46.672335 4832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:28:46.673560 4832 hybrid_clock.cc:648] HybridClock initialized: now 1786404526673521 us; error 37 us; skew 500 ppm
I20260810 23:28:46.673964 4832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:28:46 dist-test-slave-rmhg krb5kdc[4807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404526, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:28:46.681645 4832 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:28:46.683365 4832 webserver.cc:460] Webserver started at http://127.14.175.126:45543/ using document root <none> and password file <none>
I20260810 23:28:46.684094 4832 fs_manager.cc:359] Metadata directory not provided
I20260810 23:28:46.684237 4832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:28:46.684563 4832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:28:46.687779 4832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "eb2dc92dfa794feb958788b902dff102"
format_stamp: "Formatted at 2026-08-10 23:28:46 on dist-test-slave-rmhg"
server_key: "1bd56fd22eb3dbba1fed953350aff19f"
server_key_iv: "eff27a950289987b7c118f5a7d751645"
server_key_version: "encryptionkey@0"
I20260810 23:28:46.688599 4832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "eb2dc92dfa794feb958788b902dff102"
format_stamp: "Formatted at 2026-08-10 23:28:46 on dist-test-slave-rmhg"
server_key: "1bd56fd22eb3dbba1fed953350aff19f"
server_key_iv: "eff27a950289987b7c118f5a7d751645"
server_key_version: "encryptionkey@0"
I20260810 23:28:46.694267 4832 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 23:28:46.698071 4862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:46.699541 4832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:28:46.699738 4832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "eb2dc92dfa794feb958788b902dff102"
format_stamp: "Formatted at 2026-08-10 23:28:46 on dist-test-slave-rmhg"
server_key: "1bd56fd22eb3dbba1fed953350aff19f"
server_key_iv: "eff27a950289987b7c118f5a7d751645"
server_key_version: "encryptionkey@0"
I20260810 23:28:46.699944 4832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:28:46.739552 4832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:28:46.744402 4832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:28:46.744769 4832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:28:46.764416 4832 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:42557
I20260810 23:28:46.764431 4947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:42557 every 8 connection(s)
I20260810 23:28:46.765934 4832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:28:46.770007 4951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:28:46.774801 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 4832
I20260810 23:28:46.775008 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:28:46.775764 15037 external_mini_cluster.cc:1428] Setting key 31ff45f80499f19035c7bf197a85dbb5
I20260810 23:28:46.781818 4951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102: Bootstrap starting.
I20260810 23:28:46.785923 4951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102: Neither blocks nor log segments found. Creating new log.
I20260810 23:28:46.787333 4951 log.cc:824] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102: Log is configured to *not* fsync() on all Append() calls
I20260810 23:28:46.790468 4951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102: No bootstrap required, opened a new log
Aug 10 23:28:46 dist-test-slave-rmhg krb5kdc[4807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:28:46.799284 4951 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2dc92dfa794feb958788b902dff102" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 42557 } }
I20260810 23:28:46.799824 4951 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:28:46.799930 4951 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2dc92dfa794feb958788b902dff102, State: Initialized, Role: FOLLOWER
I20260810 23:28:46.800653 4951 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2dc92dfa794feb958788b902dff102" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 42557 } }
I20260810 23:28:46.800864 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:46.778012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53324 (local address 127.14.175.126:42557)
0810 23:28:46.778933 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:46.778946 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:46.778995 (+ 49us) server_negotiation.cc:407] Connection header received
0810 23:28:46.780816 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:46.780844 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:46.781332 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:46.781705 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:46.783574 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:46.785010 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:46.787024 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:46.787800 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:46.788012 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:46.792103 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:46.792112 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:46.792129 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:46.792179 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:46.795807 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:46.796742 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:46.796754 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:46.796763 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:46.796984 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:46.797483 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:46.797487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:46.797490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:46.798182 (+ 692us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:46.798542 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:46.799018 (+ 476us) server_negotiation.cc:299] Negotiation successful
0810 23:28:46.799525 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":395,"threads_started":1}
I20260810 23:28:46.800961 4951 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:28:46.801126 4951 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:28:46.801364 4951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:28:46.804351 4951 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2dc92dfa794feb958788b902dff102" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 42557 } }
I20260810 23:28:46.804958 4951 leader_election.cc:302] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb2dc92dfa794feb958788b902dff102; no voters:
I20260810 23:28:46.805634 4951 leader_election.cc:288] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:28:46.805781 4963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:28:46.806468 4963 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [term 1 LEADER]: Becoming Leader. State: Replica: eb2dc92dfa794feb958788b902dff102, State: Running, Role: LEADER
I20260810 23:28:46.807035 4963 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2dc92dfa794feb958788b902dff102" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 42557 } }
I20260810 23:28:46.807786 4951 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:28:46.811028 4964 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb2dc92dfa794feb958788b902dff102" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2dc92dfa794feb958788b902dff102" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 42557 } } }
I20260810 23:28:46.811473 4964 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [sys.catalog]: This master's current role is: LEADER
I20260810 23:28:46.812287 4976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:28:46.812242 4966 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb2dc92dfa794feb958788b902dff102. Latest consensus state: current_term: 1 leader_uuid: "eb2dc92dfa794feb958788b902dff102" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2dc92dfa794feb958788b902dff102" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 42557 } } }
I20260810 23:28:46.812613 4966 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [sys.catalog]: This master's current role is: LEADER
I20260810 23:28:46.819375 4976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:28:46.828912 4976 catalog_manager.cc:1347] Generated new cluster ID: faaa53eb7bf34fd7ad248f8c6b08f82d
I20260810 23:28:46.829046 4976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:28:46.842958 4976 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:28:46.844350 4976 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:28:46.854123 4976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102: Generated new TSK 0
I20260810 23:28:46.855057 4976 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
I20260810 23:28:46.924712 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.65:0
--local_ip_for_outbound_sockets=127.14.175.65
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:42557
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:42173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:28:47.279362 4993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:28:47.279691 4993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:28:47.279840 4993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:28:47.287272 4993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:28:47.287407 4993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:28:47.287568 4993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.65
I20260810 23:28:47.297748 4993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:42173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:42557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.65
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:28:47.299167 4993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:28:47.300601 4993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:28:47.314757 5002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:47.314805 5005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:47.314810 5001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:28:47.315353 4993 server_base.cc:1034] running on GCE node
I20260810 23:28:47.316038 4993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:28:47.317045 4993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:28:47.318269 4993 hybrid_clock.cc:648] HybridClock initialized: now 1786404527318214 us; error 41 us; skew 500 ppm
I20260810 23:28:47.318601 4993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:28:47 dist-test-slave-rmhg krb5kdc[4807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404527, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:28:47.321898 4993 init.cc:377] Logged in from keytab as kudu/127.14.175.65@KRBTEST.COM (short username kudu)
I20260810 23:28:47.323473 4993 webserver.cc:460] Webserver started at http://127.14.175.65:33167/ using document root <none> and password file <none>
I20260810 23:28:47.324201 4993 fs_manager.cc:359] Metadata directory not provided
I20260810 23:28:47.324330 4993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:28:47.324649 4993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:28:47.327872 4993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/data/instance:
uuid: "c2025f4fdf524504bfe241da57e32cbf"
format_stamp: "Formatted at 2026-08-10 23:28:47 on dist-test-slave-rmhg"
server_key: "d49574e7b52e2633edc868116aa20f0c"
server_key_iv: "5c53a2d3a339317e7c4c5fd990c289b6"
server_key_version: "encryptionkey@0"
I20260810 23:28:47.328706 4993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance:
uuid: "c2025f4fdf524504bfe241da57e32cbf"
format_stamp: "Formatted at 2026-08-10 23:28:47 on dist-test-slave-rmhg"
server_key: "d49574e7b52e2633edc868116aa20f0c"
server_key_iv: "5c53a2d3a339317e7c4c5fd990c289b6"
server_key_version: "encryptionkey@0"
I20260810 23:28:47.334455 4993 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 23:28:47.338580 5018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:47.340262 4993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:28:47.340473 4993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/wal
uuid: "c2025f4fdf524504bfe241da57e32cbf"
format_stamp: "Formatted at 2026-08-10 23:28:47 on dist-test-slave-rmhg"
server_key: "d49574e7b52e2633edc868116aa20f0c"
server_key_iv: "5c53a2d3a339317e7c4c5fd990c289b6"
server_key_version: "encryptionkey@0"
I20260810 23:28:47.340736 4993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:28:47.359323 4993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:28:47.363864 4993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:28:47.364157 4993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:28:47.365218 4993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:28:47.366689 4993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:28:47.366765 4993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:47.366869 4993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:28:47.366925 4993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:47.399194 4993 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.65:35659
I20260810 23:28:47.399235 5197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.65:35659 every 8 connection(s)
I20260810 23:28:47.400494 4993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
I20260810 23:28:47.405848 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 4993
I20260810 23:28:47.406131 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance
I20260810 23:28:47.406940 15037 external_mini_cluster.cc:1428] Setting key febf5ecd9f040c19c7e2423b40882526
Aug 10 23:28:47 dist-test-slave-rmhg krb5kdc[4807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404527, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:28:47.423229 5202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:47.403150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:42679 (local address 127.14.175.126:42557)
0810 23:28:47.403600 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:47.403606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:47.404622 (+ 1016us) server_negotiation.cc:407] Connection header received
0810 23:28:47.407117 (+ 2495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:47.407124 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:47.407205 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:47.407298 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:47.409270 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:47.410027 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:47.412261 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:47.412662 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:47.412755 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:47.417075 (+ 4320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:47.417081 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:47.417084 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:47.417120 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:47.419822 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:47.420774 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:47.420781 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:47.420784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:47.420944 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:47.421390 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:47.421395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:47.421398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:47.421756 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:47.421913 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:47.422605 (+ 692us) server_negotiation.cc:299] Negotiation successful
0810 23:28:47.422854 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":221,"threads_started":1}
I20260810 23:28:47.423990 5203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:47.403643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.65:42679)
0810 23:28:47.404433 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:47.404498 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:47.406744 (+ 2246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:47.407611 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:47.407624 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:47.408236 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:28:47.409075 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:47.409103 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:47.410213 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:47.410218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:47.412126 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:47.412141 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:47.412919 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:47.412926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:47.413246 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:47.413998 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:28:47.414037 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:28:47.416808 (+ 2771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:28:47.419988 (+ 3180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:47.419999 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:47.420018 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:47.420638 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:47.421038 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:47.421042 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:47.421044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:47.421281 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:47.421868 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:28:47.421874 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:28:47.422220 (+ 346us) client_negotiation.cc:769] Sending connection context
0810 23:28:47.422447 (+ 227us) client_negotiation.cc:240] Negotiation successful
0810 23:28:47.422863 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":301,"threads_started":1}
I20260810 23:28:47.425526 5199 heartbeater.cc:347] Connected to a master server at 127.14.175.126:42557
I20260810 23:28:47.425813 5199 heartbeater.cc:464] Registering TS with master...
I20260810 23:28:47.426563 5199 heartbeater.cc:511] Master 127.14.175.126:42557 requested a full tablet report, sending...
I20260810 23:28:47.428673 4886 ts_manager.cc:194] Registered new tserver with Master: c2025f4fdf524504bfe241da57e32cbf (127.14.175.65:35659)
I20260810 23:28:47.430190 4886 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.65@KRBTEST.COM'} at 127.14.175.65:42679
WARNING: no policy specified for kudu/127.14.175.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
I20260810 23:28:47.468448 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.66:0
--local_ip_for_outbound_sockets=127.14.175.66
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:42557
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:42173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:28:47.824183 5206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:28:47.824533 5206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:28:47.824779 5206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:28:47.832471 5206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:28:47.832626 5206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:28:47.832787 5206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.66
I20260810 23:28:47.843000 5206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:42173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:42557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.66
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:28:47.844389 5206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:28:47.845844 5206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:28:47.858946 5217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:47.858882 5218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:47.858924 5221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:28:47.859709 5206 server_base.cc:1034] running on GCE node
I20260810 23:28:47.860289 5206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:28:47.861275 5206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:28:47.862519 5206 hybrid_clock.cc:648] HybridClock initialized: now 1786404527862459 us; error 47 us; skew 500 ppm
I20260810 23:28:47.862895 5206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:28:47 dist-test-slave-rmhg krb5kdc[4807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404527, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:28:47.866189 5206 init.cc:377] Logged in from keytab as kudu/127.14.175.66@KRBTEST.COM (short username kudu)
I20260810 23:28:47.867700 5206 webserver.cc:460] Webserver started at http://127.14.175.66:41887/ using document root <none> and password file <none>
I20260810 23:28:47.868392 5206 fs_manager.cc:359] Metadata directory not provided
I20260810 23:28:47.868516 5206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:28:47.868829 5206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:28:47.871887 5206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/data/instance:
uuid: "030a6b9f073b4e8b827b8d8d166cf958"
format_stamp: "Formatted at 2026-08-10 23:28:47 on dist-test-slave-rmhg"
server_key: "59f1ec654178333a512a97c89c12d1d7"
server_key_iv: "e5686c028fa802da1c7dabdd9325c175"
server_key_version: "encryptionkey@0"
I20260810 23:28:47.872663 5206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance:
uuid: "030a6b9f073b4e8b827b8d8d166cf958"
format_stamp: "Formatted at 2026-08-10 23:28:47 on dist-test-slave-rmhg"
server_key: "59f1ec654178333a512a97c89c12d1d7"
server_key_iv: "e5686c028fa802da1c7dabdd9325c175"
server_key_version: "encryptionkey@0"
I20260810 23:28:47.877983 5206 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 23:28:47.881649 5230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:47.883056 5206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 23:28:47.883224 5206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/wal
uuid: "030a6b9f073b4e8b827b8d8d166cf958"
format_stamp: "Formatted at 2026-08-10 23:28:47 on dist-test-slave-rmhg"
server_key: "59f1ec654178333a512a97c89c12d1d7"
server_key_iv: "e5686c028fa802da1c7dabdd9325c175"
server_key_version: "encryptionkey@0"
I20260810 23:28:47.883419 5206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:28:47.897550 5206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:28:47.902060 5206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:28:47.902374 5206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:28:47.903409 5206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:28:47.904991 5206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:28:47.905067 5206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:47.905206 5206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:28:47.905280 5206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260810 23:28:47.936767 5206 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.66:44945
I20260810 23:28:47.936808 5395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.66:44945 every 8 connection(s)
I20260810 23:28:47.938239 5206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
I20260810 23:28:47.947408 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 5206
I20260810 23:28:47.947579 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance
I20260810 23:28:47.947992 15037 external_mini_cluster.cc:1428] Setting key 73dbc64f6b5219107b00bde2b638fbfd
Aug 10 23:28:47 dist-test-slave-rmhg krb5kdc[4807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404527, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:28:47.959244 5400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:47.940968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:49199 (local address 127.14.175.126:42557)
0810 23:28:47.941466 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:47.941471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:47.942503 (+ 1032us) server_negotiation.cc:407] Connection header received
0810 23:28:47.944373 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:47.944378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:47.944482 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:47.944583 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:47.946322 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:47.947153 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:47.948920 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:47.949504 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:47.949604 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:47.953109 (+ 3505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:47.953116 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:47.953120 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:47.953194 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:47.956164 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:47.956888 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:47.956895 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:47.956899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:47.957060 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:47.957497 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:47.957502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:47.957504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:47.957842 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:47.957988 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:47.958663 (+ 675us) server_negotiation.cc:299] Negotiation successful
0810 23:28:47.958881 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":264,"threads_started":1}
I20260810 23:28:47.960129 5401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:47.941579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.66:49199)
0810 23:28:47.942327 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:47.942378 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:47.944055 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:47.944867 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:47.944878 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:47.945501 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:28:47.946119 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:47.946138 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:47.947335 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:47.947340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:47.948763 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:47.948776 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:47.949742 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:47.949750 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:47.949968 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:47.950656 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:28:47.950678 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:28:47.952869 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:28:47.956323 (+ 3454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:47.956330 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:47.956344 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:47.956767 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:47.957189 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:47.957194 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:47.957197 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:47.957394 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:47.957946 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:28:47.957952 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:28:47.958299 (+ 347us) client_negotiation.cc:769] Sending connection context
0810 23:28:47.958525 (+ 226us) client_negotiation.cc:240] Negotiation successful
0810 23:28:47.958939 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":284,"threads_started":1}
I20260810 23:28:47.961755 5396 heartbeater.cc:347] Connected to a master server at 127.14.175.126:42557
I20260810 23:28:47.962064 5396 heartbeater.cc:464] Registering TS with master...
I20260810 23:28:47.962834 5396 heartbeater.cc:511] Master 127.14.175.126:42557 requested a full tablet report, sending...
I20260810 23:28:47.964399 4886 ts_manager.cc:194] Registered new tserver with Master: 030a6b9f073b4e8b827b8d8d166cf958 (127.14.175.66:44945)
I20260810 23:28:47.965530 4886 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.66@KRBTEST.COM'} at 127.14.175.66:49199
WARNING: no policy specified for kudu/127.14.175.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
I20260810 23:28:48.009999 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.67:0
--local_ip_for_outbound_sockets=127.14.175.67
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:42557
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:42173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:28:48.356194 5408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:28:48.356511 5408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:28:48.356617 5408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:28:48.363929 5408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:28:48.364056 5408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:28:48.364172 5408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.67
I20260810 23:28:48.374253 5408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:42173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:42557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.67
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:28:48.375716 5408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:28:48.377555 5408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:28:48.393697 5416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:48.393736 5417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:48.393764 5419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:28:48.394505 5408 server_base.cc:1034] running on GCE node
I20260810 23:28:48.395195 5408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:28:48.396162 5408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:28:48.397413 5408 hybrid_clock.cc:648] HybridClock initialized: now 1786404528397354 us; error 43 us; skew 500 ppm
I20260810 23:28:48.397784 5408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:28:48 dist-test-slave-rmhg krb5kdc[4807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404528, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:28:48.401785 5408 init.cc:377] Logged in from keytab as kudu/127.14.175.67@KRBTEST.COM (short username kudu)
I20260810 23:28:48.403558 5408 webserver.cc:460] Webserver started at http://127.14.175.67:33487/ using document root <none> and password file <none>
I20260810 23:28:48.404484 5408 fs_manager.cc:359] Metadata directory not provided
I20260810 23:28:48.404630 5408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:28:48.404978 5408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:28:48.407982 5408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/data/instance:
uuid: "0ae6b985366f4d47914c40302b102253"
format_stamp: "Formatted at 2026-08-10 23:28:48 on dist-test-slave-rmhg"
server_key: "a60d61e517106eb7f148903bc164cc02"
server_key_iv: "20126f69f44dfc76e4e820faf3c1c8d4"
server_key_version: "encryptionkey@0"
I20260810 23:28:48.408752 5408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance:
uuid: "0ae6b985366f4d47914c40302b102253"
format_stamp: "Formatted at 2026-08-10 23:28:48 on dist-test-slave-rmhg"
server_key: "a60d61e517106eb7f148903bc164cc02"
server_key_iv: "20126f69f44dfc76e4e820faf3c1c8d4"
server_key_version: "encryptionkey@0"
I20260810 23:28:48.414124 5408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 23:28:48.417825 5429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:48.419202 5408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:28:48.419359 5408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/wal
uuid: "0ae6b985366f4d47914c40302b102253"
format_stamp: "Formatted at 2026-08-10 23:28:48 on dist-test-slave-rmhg"
server_key: "a60d61e517106eb7f148903bc164cc02"
server_key_iv: "20126f69f44dfc76e4e820faf3c1c8d4"
server_key_version: "encryptionkey@0"
I20260810 23:28:48.419550 5408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:28:48.432572 5408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:28:48.433116 5199 heartbeater.cc:503] Master 127.14.175.126:42557 was elected leader, sending a full tablet report...
I20260810 23:28:48.437036 5408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:28:48.437328 5408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:28:48.438297 5408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:28:48.439651 5408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:28:48.439739 5408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:48.439863 5408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:28:48.439917 5408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:48.470634 5408 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.67:37491
I20260810 23:28:48.470693 5606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.67:37491 every 8 connection(s)
I20260810 23:28:48.471926 5408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
I20260810 23:28:48.475127 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 5408
I20260810 23:28:48.475353 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance
I20260810 23:28:48.475934 15037 external_mini_cluster.cc:1428] Setting key 8c274bcf3d3a449ddb62ba11eb4ee628
Aug 10 23:28:48 dist-test-slave-rmhg krb5kdc[4807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404528, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:28:48.492656 5610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:48.474446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:46595 (local address 127.14.175.126:42557)
0810 23:28:48.474924 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:48.474929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:48.475958 (+ 1029us) server_negotiation.cc:407] Connection header received
0810 23:28:48.478060 (+ 2102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:48.478066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:48.478164 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:48.478260 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:48.479981 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:48.480792 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:48.482440 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:48.482863 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:48.482979 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:48.486619 (+ 3640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:48.486624 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:48.486627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:48.486666 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:48.489272 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:48.490171 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:48.490177 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:48.490180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:48.490342 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:48.490802 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:48.490808 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:48.490811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:48.491106 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:48.491268 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:48.492025 (+ 757us) server_negotiation.cc:299] Negotiation successful
0810 23:28:48.492282 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":256,"threads_started":1}
I20260810 23:28:48.493709 5612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:48.474971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.67:46595)
0810 23:28:48.475788 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:48.475830 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:48.477735 (+ 1905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:48.478524 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:48.478533 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:48.479178 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:28:48.479802 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:48.479822 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:48.480938 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:48.480942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:48.482287 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:48.482301 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:48.483097 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:48.483105 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:48.483340 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:48.483968 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:28:48.483991 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:28:48.486360 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:28:48.489430 (+ 3070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:48.489440 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:48.489457 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:48.490020 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:48.490455 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:48.490460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:48.490464 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:48.490678 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:48.491248 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:28:48.491257 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:28:48.491617 (+ 360us) client_negotiation.cc:769] Sending connection context
0810 23:28:48.491884 (+ 267us) client_negotiation.cc:240] Negotiation successful
0810 23:28:48.492467 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":318,"threads_started":1}
I20260810 23:28:48.495127 5608 heartbeater.cc:347] Connected to a master server at 127.14.175.126:42557
I20260810 23:28:48.495431 5608 heartbeater.cc:464] Registering TS with master...
I20260810 23:28:48.496177 5608 heartbeater.cc:511] Master 127.14.175.126:42557 requested a full tablet report, sending...
I20260810 23:28:48.497699 4886 ts_manager.cc:194] Registered new tserver with Master: 0ae6b985366f4d47914c40302b102253 (127.14.175.67:37491)
I20260810 23:28:48.498510 4886 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.67@KRBTEST.COM'} at 127.14.175.67:46595
I20260810 23:28:48.503609 15037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:28:48.521229 5610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:48.508241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53328 (local address 127.14.175.126:42557)
0810 23:28:48.508420 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:48.508424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:48.508922 (+ 498us) server_negotiation.cc:407] Connection header received
0810 23:28:48.509106 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:48.509112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:48.509233 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:48.509338 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:48.510677 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:48.511468 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:48.512871 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:48.513356 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:48.513467 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:48.515929 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:48.515934 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:48.515937 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:48.515971 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:48.518470 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:48.519168 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:48.519182 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:48.519186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:48.519316 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:48.519684 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:48.519689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:48.519692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:48.520042 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:48.520298 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:48.520630 (+ 332us) server_negotiation.cc:299] Negotiation successful
0810 23:28:48.520874 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 23:28:48.530174 4886 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:28:48.537349 4886 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:28:48.573228 5634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:48.550700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:44945 (local address 127.0.0.1:41568)
0810 23:28:48.564668 (+ 13968us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:48.564693 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:48.564864 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:48.565737 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:48.565741 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:48.565766 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:28:48.566178 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:48.566194 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:48.568077 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:48.568082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:48.570196 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:48.570210 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:48.572184 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:48.572189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:48.572337 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:48.572373 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 23:28:48.572489 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 23:28:48.572708 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13789,"mutex_wait_us":126,"thread_start_us":14443,"threads_started":1}
I20260810 23:28:48.573571 5630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:48.550386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52314 (local address 127.14.175.67:37491)
0810 23:28:48.563527 (+ 13141us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:48.563537 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:48.563576 (+ 39us) server_negotiation.cc:407] Connection header received
0810 23:28:48.563701 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:48.563710 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:48.564001 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:48.565328 (+ 1327us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:28:48.565619 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:48.567380 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:48.570911 (+ 3531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:48.572369 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:48.572506 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:48.572744 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:48.573003 (+ 259us) server_negotiation.cc:299] Negotiation successful
0810 23:28:48.573212 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12963,"thread_start_us":12801,"threads_started":1}
I20260810 23:28:48.574033 5637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:48.551425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58316 (local address 127.14.175.65:35659)
0810 23:28:48.564072 (+ 12647us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:48.564083 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:48.565554 (+ 1471us) server_negotiation.cc:407] Connection header received
0810 23:28:48.565704 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:48.565712 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:48.565949 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:48.566153 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:28:48.566754 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:48.568239 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:48.571102 (+ 2863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:48.572488 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:48.572606 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:48.572810 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:48.573606 (+ 796us) server_negotiation.cc:299] Negotiation successful
0810 23:28:48.573790 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12434,"thread_start_us":12239,"threads_started":1}
I20260810 23:28:48.574069 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:48.550852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41568 (local address 127.14.175.66:44945)
0810 23:28:48.561692 (+ 10840us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:48.561705 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:48.565137 (+ 3432us) server_negotiation.cc:407] Connection header received
0810 23:28:48.565281 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:48.565289 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:48.565536 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:48.565752 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:28:48.566393 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:48.567900 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:48.570363 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:48.571709 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:48.571842 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:48.572059 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:48.573641 (+ 1582us) server_negotiation.cc:299] Negotiation successful
0810 23:28:48.573821 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10605,"thread_start_us":10859,"threads_started":1}
I20260810 23:28:48.573484 5629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:48.550062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:37491 (local address 127.0.0.1:52314)
0810 23:28:48.562550 (+ 12488us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:48.562618 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:48.562860 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:48.564318 (+ 1458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:48.564326 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:48.564376 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:28:48.564949 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:48.564966 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:48.567859 (+ 2893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:48.567864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:48.570694 (+ 2830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:48.570714 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:48.572595 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:48.572601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:48.572775 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:48.572806 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 23:28:48.572904 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 23:28:48.573036 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12190,"mutex_wait_us":157,"thread_start_us":12032,"threads_started":1}
I20260810 23:28:48.574532 5632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:48.550805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:35659 (local address 127.0.0.1:58316)
0810 23:28:48.565419 (+ 14614us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:48.565435 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:48.565550 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:48.566165 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:48.566170 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:48.566197 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:28:48.566564 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:48.566575 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:48.568386 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:48.568390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:48.570730 (+ 2340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:48.570740 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:48.572961 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:48.572968 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:48.573194 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:48.573230 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 23:28:48.573316 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 23:28:48.573458 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14493,"mutex_wait_us":25,"thread_start_us":13467,"threads_started":1}
I20260810 23:28:48.578121 5506 tablet_service.cc:1467] Processing CreateTablet for tablet e0616279ce9541ef83c94b815670c399 (DEFAULT_TABLE table=test-table [id=e1cc7c32f77544f1bd4c11f8941b8a49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:28:48.578126 5295 tablet_service.cc:1467] Processing CreateTablet for tablet e0616279ce9541ef83c94b815670c399 (DEFAULT_TABLE table=test-table [id=e1cc7c32f77544f1bd4c11f8941b8a49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:28:48.579425 5295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0616279ce9541ef83c94b815670c399. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:28:48.579512 5506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0616279ce9541ef83c94b815670c399. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:28:48.581758 5095 tablet_service.cc:1467] Processing CreateTablet for tablet e0616279ce9541ef83c94b815670c399 (DEFAULT_TABLE table=test-table [id=e1cc7c32f77544f1bd4c11f8941b8a49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:28:48.583240 5095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0616279ce9541ef83c94b815670c399. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:28:48.593853 5639 tablet_bootstrap.cc:492] T e0616279ce9541ef83c94b815670c399 P 030a6b9f073b4e8b827b8d8d166cf958: Bootstrap starting.
I20260810 23:28:48.596095 5640 tablet_bootstrap.cc:492] T e0616279ce9541ef83c94b815670c399 P 0ae6b985366f4d47914c40302b102253: Bootstrap starting.
I20260810 23:28:48.598519 5642 tablet_bootstrap.cc:492] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf: Bootstrap starting.
I20260810 23:28:48.599699 5639 tablet_bootstrap.cc:654] T e0616279ce9541ef83c94b815670c399 P 030a6b9f073b4e8b827b8d8d166cf958: Neither blocks nor log segments found. Creating new log.
I20260810 23:28:48.599685 5640 tablet_bootstrap.cc:654] T e0616279ce9541ef83c94b815670c399 P 0ae6b985366f4d47914c40302b102253: Neither blocks nor log segments found. Creating new log.
I20260810 23:28:48.601064 5640 log.cc:824] T e0616279ce9541ef83c94b815670c399 P 0ae6b985366f4d47914c40302b102253: Log is configured to *not* fsync() on all Append() calls
I20260810 23:28:48.601068 5639 log.cc:824] T e0616279ce9541ef83c94b815670c399 P 030a6b9f073b4e8b827b8d8d166cf958: Log is configured to *not* fsync() on all Append() calls
I20260810 23:28:48.601631 5642 tablet_bootstrap.cc:654] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf: Neither blocks nor log segments found. Creating new log.
I20260810 23:28:48.603098 5642 log.cc:824] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf: Log is configured to *not* fsync() on all Append() calls
I20260810 23:28:48.604831 5640 tablet_bootstrap.cc:492] T e0616279ce9541ef83c94b815670c399 P 0ae6b985366f4d47914c40302b102253: No bootstrap required, opened a new log
I20260810 23:28:48.604856 5639 tablet_bootstrap.cc:492] T e0616279ce9541ef83c94b815670c399 P 030a6b9f073b4e8b827b8d8d166cf958: No bootstrap required, opened a new log
I20260810 23:28:48.605225 5639 ts_tablet_manager.cc:1397] T e0616279ce9541ef83c94b815670c399 P 030a6b9f073b4e8b827b8d8d166cf958: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20260810 23:28:48.605321 5640 ts_tablet_manager.cc:1397] T e0616279ce9541ef83c94b815670c399 P 0ae6b985366f4d47914c40302b102253: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260810 23:28:48.607905 5642 tablet_bootstrap.cc:492] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf: No bootstrap required, opened a new log
I20260810 23:28:48.608203 5642 ts_tablet_manager.cc:1397] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.002s
I20260810 23:28:48.613281 5640 raft_consensus.cc:348] T e0616279ce9541ef83c94b815670c399 P 0ae6b985366f4d47914c40302b102253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae6b985366f4d47914c40302b102253" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37491 } } peers { permanent_uuid: "030a6b9f073b4e8b827b8d8d166cf958" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44945 } } peers { permanent_uuid: "c2025f4fdf524504bfe241da57e32cbf" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35659 } }
I20260810 23:28:48.613791 5640 raft_consensus.cc:374] T e0616279ce9541ef83c94b815670c399 P 0ae6b985366f4d47914c40302b102253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:28:48.613910 5640 raft_consensus.cc:729] T e0616279ce9541ef83c94b815670c399 P 0ae6b985366f4d47914c40302b102253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ae6b985366f4d47914c40302b102253, State: Initialized, Role: FOLLOWER
I20260810 23:28:48.614862 5640 consensus_queue.cc:260] T e0616279ce9541ef83c94b815670c399 P 0ae6b985366f4d47914c40302b102253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae6b985366f4d47914c40302b102253" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37491 } } peers { permanent_uuid: "030a6b9f073b4e8b827b8d8d166cf958" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44945 } } peers { permanent_uuid: "c2025f4fdf524504bfe241da57e32cbf" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35659 } }
I20260810 23:28:48.615792 5639 raft_consensus.cc:348] T e0616279ce9541ef83c94b815670c399 P 030a6b9f073b4e8b827b8d8d166cf958 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae6b985366f4d47914c40302b102253" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37491 } } peers { permanent_uuid: "030a6b9f073b4e8b827b8d8d166cf958" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44945 } } peers { permanent_uuid: "c2025f4fdf524504bfe241da57e32cbf" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35659 } }
I20260810 23:28:48.616279 5639 raft_consensus.cc:374] T e0616279ce9541ef83c94b815670c399 P 030a6b9f073b4e8b827b8d8d166cf958 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:28:48.616377 5639 raft_consensus.cc:729] T e0616279ce9541ef83c94b815670c399 P 030a6b9f073b4e8b827b8d8d166cf958 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 030a6b9f073b4e8b827b8d8d166cf958, State: Initialized, Role: FOLLOWER
I20260810 23:28:48.617038 5639 consensus_queue.cc:260] T e0616279ce9541ef83c94b815670c399 P 030a6b9f073b4e8b827b8d8d166cf958 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae6b985366f4d47914c40302b102253" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37491 } } peers { permanent_uuid: "030a6b9f073b4e8b827b8d8d166cf958" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44945 } } peers { permanent_uuid: "c2025f4fdf524504bfe241da57e32cbf" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35659 } }
I20260810 23:28:48.618392 5396 heartbeater.cc:503] Master 127.14.175.126:42557 was elected leader, sending a full tablet report...
I20260810 23:28:48.618721 5642 raft_consensus.cc:348] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae6b985366f4d47914c40302b102253" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37491 } } peers { permanent_uuid: "030a6b9f073b4e8b827b8d8d166cf958" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44945 } } peers { permanent_uuid: "c2025f4fdf524504bfe241da57e32cbf" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35659 } }
I20260810 23:28:48.619071 5639 ts_tablet_manager.cc:1428] T e0616279ce9541ef83c94b815670c399 P 030a6b9f073b4e8b827b8d8d166cf958: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260810 23:28:48.619185 5642 raft_consensus.cc:374] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:28:48.619305 5642 raft_consensus.cc:729] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2025f4fdf524504bfe241da57e32cbf, State: Initialized, Role: FOLLOWER
I20260810 23:28:48.620075 5642 consensus_queue.cc:260] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae6b985366f4d47914c40302b102253" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37491 } } peers { permanent_uuid: "030a6b9f073b4e8b827b8d8d166cf958" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44945 } } peers { permanent_uuid: "c2025f4fdf524504bfe241da57e32cbf" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35659 } }
I20260810 23:28:48.620232 5608 heartbeater.cc:503] Master 127.14.175.126:42557 was elected leader, sending a full tablet report...
I20260810 23:28:48.621814 5642 ts_tablet_manager.cc:1428] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf: Time spent starting tablet: real 0.013s user 0.013s sys 0.002s
I20260810 23:28:48.622864 5640 ts_tablet_manager.cc:1428] T e0616279ce9541ef83c94b815670c399 P 0ae6b985366f4d47914c40302b102253: Time spent starting tablet: real 0.017s user 0.009s sys 0.006s
W20260810 23:28:48.652971 5200 tablet.cc:2369] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:28:48.667409 5654 raft_consensus.cc:482] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:28:48.667721 5654 raft_consensus.cc:504] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae6b985366f4d47914c40302b102253" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37491 } } peers { permanent_uuid: "030a6b9f073b4e8b827b8d8d166cf958" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44945 } } peers { permanent_uuid: "c2025f4fdf524504bfe241da57e32cbf" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35659 } }
I20260810 23:28:48.669567 5654 leader_election.cc:288] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ae6b985366f4d47914c40302b102253 (127.14.175.67:37491), 030a6b9f073b4e8b827b8d8d166cf958 (127.14.175.66:44945)
I20260810 23:28:48.678699 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:48.670132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:55583 (local address 127.14.175.66:44945)
0810 23:28:48.670339 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:48.670345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:48.671083 (+ 738us) server_negotiation.cc:407] Connection header received
0810 23:28:48.671272 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:48.671277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:48.671356 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:48.671458 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:28:48.672304 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:48.673453 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:48.676331 (+ 2878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:48.677576 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:48.677754 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:48.677835 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:48.678280 (+ 445us) server_negotiation.cc:299] Negotiation successful
0810 23:28:48.678417 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 23:28:48.678691 5630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:48.670028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:36599 (local address 127.14.175.67:37491)
0810 23:28:48.670242 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:48.670249 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:48.671044 (+ 795us) server_negotiation.cc:407] Connection header received
0810 23:28:48.671272 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:48.671277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:48.671356 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:48.671458 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:28:48.672304 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:48.673457 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:48.676328 (+ 2871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:48.677588 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:48.677754 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:48.677835 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:48.678280 (+ 445us) server_negotiation.cc:299] Negotiation successful
0810 23:28:48.678417 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 23:28:48.678943 5656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:48.670042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:37491 (local address 127.14.175.65:36599)
0810 23:28:48.670918 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:48.670939 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:48.671117 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:48.671480 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:48.671485 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:48.671541 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:28:48.672131 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:48.672153 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:48.673589 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:48.673594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:48.676154 (+ 2560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:48.676171 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:48.677847 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:48.677852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:48.678066 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:48.678140 (+ 74us) client_negotiation.cc:769] Sending connection context
0810 23:28:48.678249 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 23:28:48.678464 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":181,"spinlock_wait_cycles":32512,"thread_start_us":490,"threads_started":1}
I20260810 23:28:48.678946 5655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:48.670073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:44945 (local address 127.14.175.65:55583)
0810 23:28:48.670989 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:48.671003 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:48.671150 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:48.671480 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:48.671485 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:48.671510 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:28:48.672133 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:48.672145 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:48.673588 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:48.673594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:48.676151 (+ 2557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:48.676171 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:48.677827 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:48.677835 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:48.678066 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:48.678146 (+ 80us) client_negotiation.cc:769] Sending connection context
0810 23:28:48.678249 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 23:28:48.678464 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"mutex_wait_us":285,"spinlock_wait_cycles":26880,"thread_start_us":449,"threads_started":1}
I20260810 23:28:48.679999 5331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0616279ce9541ef83c94b815670c399" candidate_uuid: "c2025f4fdf524504bfe241da57e32cbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "030a6b9f073b4e8b827b8d8d166cf958" is_pre_election: true
I20260810 23:28:48.679999 5538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0616279ce9541ef83c94b815670c399" candidate_uuid: "c2025f4fdf524504bfe241da57e32cbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ae6b985366f4d47914c40302b102253" is_pre_election: true
I20260810 23:28:48.680497 5538 raft_consensus.cc:2454] T e0616279ce9541ef83c94b815670c399 P 0ae6b985366f4d47914c40302b102253 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2025f4fdf524504bfe241da57e32cbf in term 0.
I20260810 23:28:48.680497 5331 raft_consensus.cc:2454] T e0616279ce9541ef83c94b815670c399 P 030a6b9f073b4e8b827b8d8d166cf958 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2025f4fdf524504bfe241da57e32cbf in term 0.
I20260810 23:28:48.681203 5020 leader_election.cc:302] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 030a6b9f073b4e8b827b8d8d166cf958, c2025f4fdf524504bfe241da57e32cbf; no voters:
I20260810 23:28:48.681622 5654 raft_consensus.cc:2781] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:28:48.681798 5654 raft_consensus.cc:482] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:28:48.681908 5654 raft_consensus.cc:3037] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:28:48.684131 5654 raft_consensus.cc:504] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae6b985366f4d47914c40302b102253" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37491 } } peers { permanent_uuid: "030a6b9f073b4e8b827b8d8d166cf958" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44945 } } peers { permanent_uuid: "c2025f4fdf524504bfe241da57e32cbf" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35659 } }
I20260810 23:28:48.684767 5654 leader_election.cc:288] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf [CANDIDATE]: Term 1 election: Requested vote from peers 0ae6b985366f4d47914c40302b102253 (127.14.175.67:37491), 030a6b9f073b4e8b827b8d8d166cf958 (127.14.175.66:44945)
I20260810 23:28:48.685364 5538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0616279ce9541ef83c94b815670c399" candidate_uuid: "c2025f4fdf524504bfe241da57e32cbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ae6b985366f4d47914c40302b102253"
I20260810 23:28:48.685365 5331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0616279ce9541ef83c94b815670c399" candidate_uuid: "c2025f4fdf524504bfe241da57e32cbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "030a6b9f073b4e8b827b8d8d166cf958"
I20260810 23:28:48.685559 5538 raft_consensus.cc:3037] T e0616279ce9541ef83c94b815670c399 P 0ae6b985366f4d47914c40302b102253 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:28:48.685568 5331 raft_consensus.cc:3037] T e0616279ce9541ef83c94b815670c399 P 030a6b9f073b4e8b827b8d8d166cf958 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:28:48.687906 5538 raft_consensus.cc:2454] T e0616279ce9541ef83c94b815670c399 P 0ae6b985366f4d47914c40302b102253 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2025f4fdf524504bfe241da57e32cbf in term 1.
I20260810 23:28:48.688088 5331 raft_consensus.cc:2454] T e0616279ce9541ef83c94b815670c399 P 030a6b9f073b4e8b827b8d8d166cf958 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2025f4fdf524504bfe241da57e32cbf in term 1.
I20260810 23:28:48.688453 5022 leader_election.cc:302] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ae6b985366f4d47914c40302b102253, c2025f4fdf524504bfe241da57e32cbf; no voters:
I20260810 23:28:48.688822 5654 raft_consensus.cc:2781] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:28:48.689349 5654 raft_consensus.cc:686] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf [term 1 LEADER]: Becoming Leader. State: Replica: c2025f4fdf524504bfe241da57e32cbf, State: Running, Role: LEADER
I20260810 23:28:48.690006 5654 consensus_queue.cc:237] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae6b985366f4d47914c40302b102253" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37491 } } peers { permanent_uuid: "030a6b9f073b4e8b827b8d8d166cf958" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44945 } } peers { permanent_uuid: "c2025f4fdf524504bfe241da57e32cbf" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35659 } }
W20260810 23:28:48.690255 5398 tablet.cc:2369] T e0616279ce9541ef83c94b815670c399 P 030a6b9f073b4e8b827b8d8d166cf958: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:28:48.695098 4886 catalog_manager.cc:5495] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf reported cstate change: term changed from 0 to 1, leader changed from <none> to c2025f4fdf524504bfe241da57e32cbf (127.14.175.65). New cstate: current_term: 1 leader_uuid: "c2025f4fdf524504bfe241da57e32cbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae6b985366f4d47914c40302b102253" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "030a6b9f073b4e8b827b8d8d166cf958" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2025f4fdf524504bfe241da57e32cbf" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35659 } health_report { overall_health: HEALTHY } } }
W20260810 23:28:48.723552 5609 tablet.cc:2369] T e0616279ce9541ef83c94b815670c399 P 0ae6b985366f4d47914c40302b102253: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:28:48.732585 5637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:48.727211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58326 (local address 127.14.175.65:35659)
0810 23:28:48.727394 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:48.727398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:48.727417 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:28:48.727573 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:48.727577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:48.727661 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:48.727801 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:28:48.728188 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:48.728986 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:48.730514 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:48.730956 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:48.731060 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:48.731582 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:28:48.732138 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:28:48.732253 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:48.732295 (+ 42us) server_negotiation.cc:299] Negotiation successful
0810 23:28:48.732368 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:28:48.741240 5331 raft_consensus.cc:1261] T e0616279ce9541ef83c94b815670c399 P 030a6b9f073b4e8b827b8d8d166cf958 [term 1 FOLLOWER]: Refusing update from remote peer c2025f4fdf524504bfe241da57e32cbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:28:48.741262 5538 raft_consensus.cc:1261] T e0616279ce9541ef83c94b815670c399 P 0ae6b985366f4d47914c40302b102253 [term 1 FOLLOWER]: Refusing update from remote peer c2025f4fdf524504bfe241da57e32cbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:28:48.742341 5657 consensus_queue.cc:1035] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "030a6b9f073b4e8b827b8d8d166cf958" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:28:48.742713 5654 consensus_queue.cc:1035] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ae6b985366f4d47914c40302b102253" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:28:48.754853 5662 mvcc.cc:205] Tried to move back new op lower bound from 7317112949713584128 to 7317112949518016512. Current Snapshot: MvccSnapshot[applied={T|T < 7317112949713584128}]
I20260810 23:28:48.757365 5663 mvcc.cc:205] Tried to move back new op lower bound from 7317112949713584128 to 7317112949518016512. Current Snapshot: MvccSnapshot[applied={T|T < 7317112949713584128}]
I20260810 23:28:48.772218 4884 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:28:48.772586 4884 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:28:48.777725 4884 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102: Sending DeleteTablet for 3 replicas of tablet e0616279ce9541ef83c94b815670c399
I20260810 23:28:48.779265 5295 tablet_service.cc:1514] Processing DeleteTablet for tablet e0616279ce9541ef83c94b815670c399 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:28:48 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:41568
I20260810 23:28:48.779286 5506 tablet_service.cc:1514] Processing DeleteTablet for tablet e0616279ce9541ef83c94b815670c399 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:28:48 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:52314
I20260810 23:28:48.785396 5095 tablet_service.cc:1514] Processing DeleteTablet for tablet e0616279ce9541ef83c94b815670c399 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:28:48 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:58316
I20260810 23:28:48.786078 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 4993
I20260810 23:28:48.793434 5678 tablet_replica.cc:330] T e0616279ce9541ef83c94b815670c399 P 030a6b9f073b4e8b827b8d8d166cf958: stopping tablet replica
I20260810 23:28:48.794025 5678 raft_consensus.cc:2229] T e0616279ce9541ef83c94b815670c399 P 030a6b9f073b4e8b827b8d8d166cf958 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:28:48.794631 5678 raft_consensus.cc:2258] T e0616279ce9541ef83c94b815670c399 P 030a6b9f073b4e8b827b8d8d166cf958 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:28:48.795549 5680 tablet_replica.cc:330] T e0616279ce9541ef83c94b815670c399 P 0ae6b985366f4d47914c40302b102253: stopping tablet replica
I20260810 23:28:48.796072 5680 raft_consensus.cc:2229] T e0616279ce9541ef83c94b815670c399 P 0ae6b985366f4d47914c40302b102253 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:28:48.796609 5680 raft_consensus.cc:2258] T e0616279ce9541ef83c94b815670c399 P 0ae6b985366f4d47914c40302b102253 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:28:48.799175 5680 ts_tablet_manager.cc:1905] T e0616279ce9541ef83c94b815670c399 P 0ae6b985366f4d47914c40302b102253: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 23:28:48 dist-test-slave-rmhg krb5kdc[4807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.65@KRBTEST.COM
I20260810 23:28:48.802742 5681 tablet_replica.cc:330] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf: stopping tablet replica
I20260810 23:28:48.803366 5681 raft_consensus.cc:2229] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:28:48.805898 5678 ts_tablet_manager.cc:1905] T e0616279ce9541ef83c94b815670c399 P 030a6b9f073b4e8b827b8d8d166cf958: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:28:48.807331 5680 ts_tablet_manager.cc:1918] T e0616279ce9541ef83c94b815670c399 P 0ae6b985366f4d47914c40302b102253: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:28:48.807507 5680 log.cc:1196] T e0616279ce9541ef83c94b815670c399 P 0ae6b985366f4d47914c40302b102253: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/wal/wals/e0616279ce9541ef83c94b815670c399
I20260810 23:28:48.808171 5680 ts_tablet_manager.cc:1939] T e0616279ce9541ef83c94b815670c399 P 0ae6b985366f4d47914c40302b102253: Deleting consensus metadata
I20260810 23:28:48.804235 5681 raft_consensus.cc:2258] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:28:48.810276 5637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:48.787415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58342 (local address 127.14.175.65:35659)
0810 23:28:48.787637 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:48.787644 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:48.787837 (+ 193us) server_negotiation.cc:407] Connection header received
0810 23:28:48.788166 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:48.788171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:48.788275 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:48.788358 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:48.790608 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:48.791922 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:48.793893 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:48.794596 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:48.794703 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:48.800548 (+ 5845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:48.800567 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:48.800589 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:48.800640 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:48.805709 (+ 5069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:48.806783 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:48.806793 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:48.806796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:48.806995 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:48.807550 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:48.807559 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:48.807562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:48.808089 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:48.808402 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:48.809620 (+ 1218us) server_negotiation.cc:299] Negotiation successful
0810 23:28:48.809904 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 23:28:48.810628 4867 catalog_manager.cc:4841] TS 0ae6b985366f4d47914c40302b102253 (127.14.175.67:37491): tablet e0616279ce9541ef83c94b815670c399 (table test-table [id=e1cc7c32f77544f1bd4c11f8941b8a49]) successfully deleted
I20260810 23:28:48.811921 5193 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:28:48.813510 5678 ts_tablet_manager.cc:1918] T e0616279ce9541ef83c94b815670c399 P 030a6b9f073b4e8b827b8d8d166cf958: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:28:48.813691 5678 log.cc:1196] T e0616279ce9541ef83c94b815670c399 P 030a6b9f073b4e8b827b8d8d166cf958: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/wal/wals/e0616279ce9541ef83c94b815670c399
I20260810 23:28:48.814260 5678 ts_tablet_manager.cc:1939] T e0616279ce9541ef83c94b815670c399 P 030a6b9f073b4e8b827b8d8d166cf958: Deleting consensus metadata
I20260810 23:28:48.816072 4864 catalog_manager.cc:4841] TS 030a6b9f073b4e8b827b8d8d166cf958 (127.14.175.66:44945): tablet e0616279ce9541ef83c94b815670c399 (table test-table [id=e1cc7c32f77544f1bd4c11f8941b8a49]) successfully deleted
I20260810 23:28:48.813529 5681 ts_tablet_manager.cc:1905] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:28:48.912109 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 4993
I20260810 23:28:48.936932 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 5206
W20260810 23:28:48.937485 4865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.175.65:35659 (error 108)
I20260810 23:28:48.940297 5634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:48.939819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:35659 (local address 127.0.0.1:58344)
0810 23:28:48.939995 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:48.940123 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260810 23:28:48.941390 4865 catalog_manager.cc:4584] TS c2025f4fdf524504bfe241da57e32cbf (127.14.175.65:35659): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e0616279ce9541ef83c94b815670c399: Network error: Client connection negotiation failed: client connection to 127.14.175.65:35659: connect: Connection refused (error 111)
Aug 10 23:28:48 dist-test-slave-rmhg krb5kdc[4807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.66@KRBTEST.COM
I20260810 23:28:48.954742 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:48.938148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41584 (local address 127.14.175.66:44945)
0810 23:28:48.938341 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:48.938348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:48.938368 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:28:48.938427 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:48.938432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:48.938529 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:48.938638 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:48.940297 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:48.941346 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:48.942872 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:48.943537 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:48.943705 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:48.948505 (+ 4800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:48.948522 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:48.948539 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:48.948579 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:48.951642 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:48.952534 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:48.952543 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:48.952545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:48.952676 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:48.953106 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:48.953114 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:48.953116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:48.953609 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:48.953865 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:48.954247 (+ 382us) server_negotiation.cc:299] Negotiation successful
0810 23:28:48.954434 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 23:28:48.955466 5386 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:28:48.995119 5634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:48.994534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:35659 (local address 127.0.0.1:58360)
0810 23:28:48.994772 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:48.994894 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260810 23:28:48.996268 5634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:48.995841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:35659 (local address 127.0.0.1:58366)
0810 23:28:48.996016 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:48.996128 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 23:28:49.046837 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 5206
I20260810 23:28:49.056962 5634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:49.056339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:35659 (local address 127.0.0.1:58372)
0810 23:28:49.056603 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:49.056724 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260810 23:28:49.058053 5634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:49.057645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:35659 (local address 127.0.0.1:58376)
0810 23:28:49.057812 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:49.057900 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 23:28:49.070842 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 5408
Aug 10 23:28:49 dist-test-slave-rmhg krb5kdc[4807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.67@KRBTEST.COM
I20260810 23:28:49.086524 5630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:49.071641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52320 (local address 127.14.175.67:37491)
0810 23:28:49.071854 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:49.071859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:49.071881 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:28:49.071948 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:49.071952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:49.072031 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:49.072121 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:49.073544 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:49.074289 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:49.075675 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:49.076102 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:49.076217 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:49.080785 (+ 4568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:49.080803 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:49.080823 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:49.080863 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:49.083514 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:49.084309 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:49.084317 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:49.084320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:49.084496 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:49.084889 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:49.084893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:49.084896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:49.085330 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:49.085636 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:49.085959 (+ 323us) server_negotiation.cc:299] Negotiation successful
0810 23:28:49.086205 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 23:28:49.087236 5601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:28:49.139251 5634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:49.138618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:35659 (local address 127.0.0.1:58378)
0810 23:28:49.138906 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:49.139031 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260810 23:28:49.140514 5634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:49.140029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:35659 (local address 127.0.0.1:58384)
0810 23:28:49.140267 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:49.140366 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260810 23:28:49.177680 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 5408
I20260810 23:28:49.201913 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 4832
I20260810 23:28:49.203199 4941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:28:49.316514 5634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:49.315906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:35659 (local address 127.0.0.1:58386)
0810 23:28:49.316118 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:49.316258 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 23:28:49.316534 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 4832
I20260810 23:28:49.348508 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:42557
--webserver_interface=127.14.175.126
--webserver_port=45543
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:42173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:42557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:28:49.696640 5693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:28:49.696987 5693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:28:49.697121 5693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:28:49.705084 5693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:28:49.705259 5693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:28:49.705343 5693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:28:49.705427 5693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:28:49.705493 5693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:28:49.716025 5693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:42173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:42557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:42557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=45543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:28:49.717480 5693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:28:49.718883 5693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:28:49.745461 5703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:49.745520 5708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:49.745461 5701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:28:49.750716 5693 server_base.cc:1034] running on GCE node
I20260810 23:28:49.751667 5693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:28:49.753403 5693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:28:49.754624 5693 hybrid_clock.cc:648] HybridClock initialized: now 1786404529754560 us; error 51 us; skew 500 ppm
I20260810 23:28:49.755064 5693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:28:49 dist-test-slave-rmhg krb5kdc[4807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404529, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:28:49.758865 5693 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:28:49.760617 5693 webserver.cc:460] Webserver started at http://127.14.175.126:45543/ using document root <none> and password file <none>
I20260810 23:28:49.761471 5693 fs_manager.cc:359] Metadata directory not provided
I20260810 23:28:49.761598 5693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:28:49.767958 5693 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260810 23:28:49.771385 5718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:49.772997 5693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:28:49.773231 5693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "eb2dc92dfa794feb958788b902dff102"
format_stamp: "Formatted at 2026-08-10 23:28:46 on dist-test-slave-rmhg"
server_key: "1bd56fd22eb3dbba1fed953350aff19f"
server_key_iv: "eff27a950289987b7c118f5a7d751645"
server_key_version: "encryptionkey@0"
I20260810 23:28:49.774415 5693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:28:49.797703 5693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:28:49.802220 5693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:28:49.802512 5693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:28:49.822863 5693 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:42557
I20260810 23:28:49.822887 5812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:42557 every 8 connection(s)
I20260810 23:28:49.824509 5693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:28:49.826515 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 5693
I20260810 23:28:49.827617 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.65:35659
--local_ip_for_outbound_sockets=127.14.175.65
--tserver_master_addrs=127.14.175.126:42557
--webserver_port=33167
--webserver_interface=127.14.175.65
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:42173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:28:49.834064 5813 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:28:49.837836 5813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102: Bootstrap starting.
I20260810 23:28:49.859951 5813 log.cc:824] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102: Log is configured to *not* fsync() on all Append() calls
I20260810 23:28:49.873234 5813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:28:49.873808 5813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102: Bootstrap complete.
I20260810 23:28:49.883673 5813 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2dc92dfa794feb958788b902dff102" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 42557 } }
I20260810 23:28:49.884637 5813 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2dc92dfa794feb958788b902dff102, State: Initialized, Role: FOLLOWER
I20260810 23:28:49.885365 5813 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2dc92dfa794feb958788b902dff102" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 42557 } }
I20260810 23:28:49.885627 5813 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:28:49.885759 5813 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:28:49.885931 5813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:28:49.890254 5813 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2dc92dfa794feb958788b902dff102" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 42557 } }
I20260810 23:28:49.890866 5813 leader_election.cc:302] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb2dc92dfa794feb958788b902dff102; no voters:
I20260810 23:28:49.891511 5813 leader_election.cc:288] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 23:28:49.891669 5820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 23:28:49.892920 5820 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [term 2 LEADER]: Becoming Leader. State: Replica: eb2dc92dfa794feb958788b902dff102, State: Running, Role: LEADER
I20260810 23:28:49.893561 5820 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2dc92dfa794feb958788b902dff102" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 42557 } }
I20260810 23:28:49.894024 5813 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:28:49.896521 5823 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb2dc92dfa794feb958788b902dff102. Latest consensus state: current_term: 2 leader_uuid: "eb2dc92dfa794feb958788b902dff102" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2dc92dfa794feb958788b902dff102" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 42557 } } }
I20260810 23:28:49.896950 5823 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [sys.catalog]: This master's current role is: LEADER
I20260810 23:28:49.897392 5821 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "eb2dc92dfa794feb958788b902dff102" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2dc92dfa794feb958788b902dff102" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 42557 } } }
I20260810 23:28:49.897729 5821 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [sys.catalog]: This master's current role is: LEADER
I20260810 23:28:49.913937 5829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:28:49.922118 5829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:28:49.923593 5829 catalog_manager.cc:1259] Loaded cluster ID: faaa53eb7bf34fd7ad248f8c6b08f82d
I20260810 23:28:49.923718 5829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:28:49.944420 5829 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 23:28:49.962337 5829 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 23:28:49.972115 5829 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:28:49.973987 5829 catalog_manager.cc:5879] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102: Loaded TSK: 0
I20260810 23:28:49.975749 5829 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 23:28:50.217769 5816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:28:50.218106 5816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:28:50.218252 5816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:28:50.226428 5816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:28:50.226572 5816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:28:50.226737 5816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.65
I20260810 23:28:50.237075 5816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:42173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.65:35659
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.175.65
--webserver_port=33167
--tserver_master_addrs=127.14.175.126:42557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.65
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:28:50.238515 5816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:28:50.239935 5816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:28:50.254192 5857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:50.254482 5858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:50.254915 5862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:28:50.256376 5816 server_base.cc:1034] running on GCE node
I20260810 23:28:50.257212 5816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:28:50.258500 5816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:28:50.259763 5816 hybrid_clock.cc:648] HybridClock initialized: now 1786404530259685 us; error 62 us; skew 500 ppm
I20260810 23:28:50.260185 5816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:28:50 dist-test-slave-rmhg krb5kdc[4807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404530, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:28:50.264276 5816 init.cc:377] Logged in from keytab as kudu/127.14.175.65@KRBTEST.COM (short username kudu)
I20260810 23:28:50.266042 5816 webserver.cc:460] Webserver started at http://127.14.175.65:33167/ using document root <none> and password file <none>
I20260810 23:28:50.266933 5816 fs_manager.cc:359] Metadata directory not provided
I20260810 23:28:50.267081 5816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:28:50.274600 5816 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 23:28:50.278695 5873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:50.280599 5816 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260810 23:28:50.280792 5816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/wal
uuid: "c2025f4fdf524504bfe241da57e32cbf"
format_stamp: "Formatted at 2026-08-10 23:28:47 on dist-test-slave-rmhg"
server_key: "d49574e7b52e2633edc868116aa20f0c"
server_key_iv: "5c53a2d3a339317e7c4c5fd990c289b6"
server_key_version: "encryptionkey@0"
I20260810 23:28:50.282217 5816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:28:50.316834 5816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:28:50.322441 5816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:28:50.322705 5816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:28:50.323784 5816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:28:50.325861 5884 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260810 23:28:50.332527 5816 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 23:28:50.332688 5816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260810 23:28:50.332834 5816 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 23:28:50.336472 5816 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 23:28:50.336581 5816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.001s sys 0.000s
I20260810 23:28:50.337095 5884 tablet_bootstrap.cc:492] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf: Bootstrap starting.
I20260810 23:28:50.347047 5884 log.cc:824] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf: Log is configured to *not* fsync() on all Append() calls
I20260810 23:28:50.366492 5884 tablet_bootstrap.cc:492] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:28:50.367590 5884 tablet_bootstrap.cc:492] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf: Bootstrap complete.
I20260810 23:28:50.368325 5884 ts_tablet_manager.cc:1397] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf: Time spent bootstrapping tablet: real 0.032s user 0.029s sys 0.000s
I20260810 23:28:50.373075 5816 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.65:35659
I20260810 23:28:50.373114 6050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.65:35659 every 8 connection(s)
I20260810 23:28:50.374811 5816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
I20260810 23:28:50.379611 5884 raft_consensus.cc:348] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae6b985366f4d47914c40302b102253" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37491 } } peers { permanent_uuid: "030a6b9f073b4e8b827b8d8d166cf958" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44945 } } peers { permanent_uuid: "c2025f4fdf524504bfe241da57e32cbf" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35659 } }
I20260810 23:28:50.381214 5884 raft_consensus.cc:729] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2025f4fdf524504bfe241da57e32cbf, State: Initialized, Role: FOLLOWER
I20260810 23:28:50.382081 5884 consensus_queue.cc:260] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae6b985366f4d47914c40302b102253" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37491 } } peers { permanent_uuid: "030a6b9f073b4e8b827b8d8d166cf958" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44945 } } peers { permanent_uuid: "c2025f4fdf524504bfe241da57e32cbf" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35659 } }
I20260810 23:28:50.382342 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 5816
I20260810 23:28:50.383162 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.66:44945
--local_ip_for_outbound_sockets=127.14.175.66
--tserver_master_addrs=127.14.175.126:42557
--webserver_port=41887
--webserver_interface=127.14.175.66
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:42173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:28:50.404402 5884 ts_tablet_manager.cc:1428] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf: Time spent starting tablet: real 0.036s user 0.030s sys 0.006s
Aug 10 23:28:50 dist-test-slave-rmhg krb5kdc[4807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404530, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:28:50.424033 6055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:50.381853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.65:52687)
0810 23:28:50.402090 (+ 20237us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:50.402215 (+ 125us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:50.403111 (+ 896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:50.405090 (+ 1979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:50.405104 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:50.406074 (+ 970us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:28:50.407020 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:50.407050 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:50.408722 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:50.408728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:50.410864 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:50.410886 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:50.411970 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:50.411977 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:50.412297 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:50.413070 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:28:50.413107 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:28:50.415889 (+ 2782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:28:50.419511 (+ 3622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:50.419521 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:50.419536 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:50.420095 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:50.420615 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:50.420620 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:50.420623 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:50.420830 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:50.421973 (+ 1143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:28:50.421979 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:28:50.422321 (+ 342us) client_negotiation.cc:769] Sending connection context
0810 23:28:50.422580 (+ 259us) client_negotiation.cc:240] Negotiation successful
0810 23:28:50.423002 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19747,"thread_start_us":20011,"threads_started":1}
I20260810 23:28:50.424582 6054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:50.380098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:52687 (local address 127.14.175.126:42557)
0810 23:28:50.401808 (+ 21710us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:50.401829 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:50.402423 (+ 594us) server_negotiation.cc:407] Connection header received
0810 23:28:50.403807 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:50.403829 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:50.404349 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:50.404816 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:50.407236 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:50.408518 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:50.411023 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:50.411754 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:50.411916 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:50.416223 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:50.416231 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:50.416250 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:50.416299 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:50.419315 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:50.420274 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:50.420283 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:50.420290 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:50.420486 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:50.421014 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:50.421021 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:50.421025 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:50.421824 (+ 799us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:50.422039 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:50.422882 (+ 843us) server_negotiation.cc:299] Negotiation successful
0810 23:28:50.423369 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21366,"thread_start_us":21206,"threads_started":1}
I20260810 23:28:50.427383 6051 heartbeater.cc:347] Connected to a master server at 127.14.175.126:42557
I20260810 23:28:50.427688 6051 heartbeater.cc:464] Registering TS with master...
I20260810 23:28:50.428422 6051 heartbeater.cc:511] Master 127.14.175.126:42557 requested a full tablet report, sending...
I20260810 23:28:50.431762 5750 ts_manager.cc:194] Registered new tserver with Master: c2025f4fdf524504bfe241da57e32cbf (127.14.175.65:35659)
I20260810 23:28:50.436566 5750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.65@KRBTEST.COM'} at 127.14.175.65:52687
I20260810 23:28:50.441069 6051 heartbeater.cc:503] Master 127.14.175.126:42557 was elected leader, sending a full tablet report...
I20260810 23:28:50.448875 6059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:50.435693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:35659 (local address 127.0.0.1:40104)
0810 23:28:50.439059 (+ 3366us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:50.439117 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:50.439311 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:50.441424 (+ 2113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:50.441432 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:50.441467 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:28:50.441902 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:50.441920 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:50.444224 (+ 2304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:50.444229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:50.445906 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:50.445925 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:50.447905 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:50.447913 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:50.448173 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:50.448242 (+ 69us) client_negotiation.cc:769] Sending connection context
0810 23:28:50.448388 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 23:28:50.448576 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3108,"thread_start_us":3803,"threads_started":1}
I20260810 23:28:50.449026 6061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:50.435930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40104 (local address 127.14.175.65:35659)
0810 23:28:50.440622 (+ 4692us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:50.440631 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:50.440671 (+ 40us) server_negotiation.cc:407] Connection header received
0810 23:28:50.440771 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:50.440778 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:50.441094 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:50.442031 (+ 937us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:28:50.442369 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:50.444047 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:50.446121 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:50.447654 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:50.447849 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:50.448131 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:50.448466 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 23:28:50.448639 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4498,"spinlock_wait_cycles":27008,"thread_start_us":4316,"threads_started":1}
I20260810 23:28:50.450505 5955 tablet_service.cc:1514] Processing DeleteTablet for tablet e0616279ce9541ef83c94b815670c399 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:28:48 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:40104
I20260810 23:28:50.452625 6063 tablet_replica.cc:330] stopping tablet replica
I20260810 23:28:50.453038 6063 raft_consensus.cc:2229] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:28:50.453469 6063 raft_consensus.cc:2258] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:28:50.454813 6063 ts_tablet_manager.cc:1905] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:28:50.462244 6063 ts_tablet_manager.cc:1918] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:28:50.462414 6063 log.cc:1196] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/wal/wals/e0616279ce9541ef83c94b815670c399
I20260810 23:28:50.462831 6063 ts_tablet_manager.cc:1939] T e0616279ce9541ef83c94b815670c399 P c2025f4fdf524504bfe241da57e32cbf: Deleting consensus metadata
I20260810 23:28:50.465122 5723 catalog_manager.cc:4841] TS c2025f4fdf524504bfe241da57e32cbf (127.14.175.65:35659): tablet e0616279ce9541ef83c94b815670c399 (table test-table [id=e1cc7c32f77544f1bd4c11f8941b8a49]) successfully deleted
W20260810 23:28:50.749912 6057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:28:50.750242 6057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:28:50.750388 6057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:28:50.757822 6057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:28:50.757972 6057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:28:50.758147 6057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.66
I20260810 23:28:50.768877 6057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:42173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.66:44945
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.175.66
--webserver_port=41887
--tserver_master_addrs=127.14.175.126:42557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.66
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:28:50.770315 6057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:28:50.771764 6057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:28:50.786203 6070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:50.786129 6075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:50.786178 6071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:28:50.787180 6057 server_base.cc:1034] running on GCE node
I20260810 23:28:50.787875 6057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:28:50.789096 6057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:28:50.790344 6057 hybrid_clock.cc:648] HybridClock initialized: now 1786404530790278 us; error 65 us; skew 500 ppm
I20260810 23:28:50.790697 6057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:28:50 dist-test-slave-rmhg krb5kdc[4807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404530, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:28:50.794140 6057 init.cc:377] Logged in from keytab as kudu/127.14.175.66@KRBTEST.COM (short username kudu)
I20260810 23:28:50.795887 6057 webserver.cc:460] Webserver started at http://127.14.175.66:41887/ using document root <none> and password file <none>
I20260810 23:28:50.796661 6057 fs_manager.cc:359] Metadata directory not provided
I20260810 23:28:50.796815 6057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:28:50.802743 6057 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260810 23:28:50.805744 6084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:50.807507 6057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 23:28:50.807719 6057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/wal
uuid: "030a6b9f073b4e8b827b8d8d166cf958"
format_stamp: "Formatted at 2026-08-10 23:28:47 on dist-test-slave-rmhg"
server_key: "59f1ec654178333a512a97c89c12d1d7"
server_key_iv: "e5686c028fa802da1c7dabdd9325c175"
server_key_version: "encryptionkey@0"
I20260810 23:28:50.808809 6057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:28:50.828804 6057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:28:50.834300 6057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:28:50.834597 6057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:28:50.835777 6057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:28:50.837309 6057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:28:50.837432 6057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:50.837548 6057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:28:50.837620 6057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:50.869807 6057 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.66:44945
I20260810 23:28:50.869880 6255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.66:44945 every 8 connection(s)
I20260810 23:28:50.871359 6057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
I20260810 23:28:50.880960 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 6057
I20260810 23:28:50.881768 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.67:37491
--local_ip_for_outbound_sockets=127.14.175.67
--tserver_master_addrs=127.14.175.126:42557
--webserver_port=33487
--webserver_interface=127.14.175.67
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:42173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:28:50 dist-test-slave-rmhg krb5kdc[4807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404530, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:28:50.894310 6054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:50.874043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:46369 (local address 127.14.175.126:42557)
0810 23:28:50.874290 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:50.874297 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:50.875638 (+ 1341us) server_negotiation.cc:407] Connection header received
0810 23:28:50.877546 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:50.877552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:50.877655 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:50.877756 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:50.879539 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:50.880322 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:50.882789 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:50.883260 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:50.883368 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:50.887880 (+ 4512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:50.887886 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:50.887889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:50.887931 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:50.890725 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:50.891669 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:50.891673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:50.891675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:50.891818 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:50.892294 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:50.892297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:50.892299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:50.892647 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:50.892816 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:50.893602 (+ 786us) server_negotiation.cc:299] Negotiation successful
0810 23:28:50.893951 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 23:28:50.895113 6262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:50.874663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.66:46369)
0810 23:28:50.875441 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:50.875494 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:50.877247 (+ 1753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:50.877993 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:50.878002 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:50.878657 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:28:50.879339 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:50.879368 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:50.880501 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:50.880506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:50.882589 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:50.882609 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:50.883538 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:50.883546 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:50.883858 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:50.884668 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:28:50.884708 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:28:50.887598 (+ 2890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:28:50.890903 (+ 3305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:50.890914 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:50.890935 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:50.891551 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:50.891922 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:50.891926 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:50.891930 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:50.892199 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:50.892775 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:28:50.892781 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:28:50.893188 (+ 407us) client_negotiation.cc:769] Sending connection context
0810 23:28:50.893450 (+ 262us) client_negotiation.cc:240] Negotiation successful
0810 23:28:50.893911 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":305,"threads_started":1}
I20260810 23:28:50.896652 6256 heartbeater.cc:347] Connected to a master server at 127.14.175.126:42557
I20260810 23:28:50.896965 6256 heartbeater.cc:464] Registering TS with master...
I20260810 23:28:50.897797 6256 heartbeater.cc:511] Master 127.14.175.126:42557 requested a full tablet report, sending...
I20260810 23:28:50.899325 5750 ts_manager.cc:194] Registered new tserver with Master: 030a6b9f073b4e8b827b8d8d166cf958 (127.14.175.66:44945)
I20260810 23:28:50.900185 5750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.66@KRBTEST.COM'} at 127.14.175.66:46369
W20260810 23:28:51.230799 6263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:28:51.231119 6263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:28:51.231253 6263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:28:51.238746 6263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:28:51.238878 6263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:28:51.239037 6263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.67
I20260810 23:28:51.249315 6263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:42173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.67:37491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.175.67
--webserver_port=33487
--tserver_master_addrs=127.14.175.126:42557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.67
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:28:51.250746 6263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:28:51.252211 6263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:28:51.266443 6275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:51.266470 6272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:51.266930 6273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:28:51.267385 6263 server_base.cc:1034] running on GCE node
I20260810 23:28:51.268062 6263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:28:51.269096 6263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:28:51.270345 6263 hybrid_clock.cc:648] HybridClock initialized: now 1786404531270289 us; error 44 us; skew 500 ppm
I20260810 23:28:51.270687 6263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:28:51 dist-test-slave-rmhg krb5kdc[4807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404531, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:28:51.273921 6263 init.cc:377] Logged in from keytab as kudu/127.14.175.67@KRBTEST.COM (short username kudu)
I20260810 23:28:51.275559 6263 webserver.cc:460] Webserver started at http://127.14.175.67:33487/ using document root <none> and password file <none>
I20260810 23:28:51.276285 6263 fs_manager.cc:359] Metadata directory not provided
I20260810 23:28:51.276417 6263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:28:51.282481 6263 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 23:28:51.285660 6285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:51.287233 6263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:28:51.287421 6263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/wal
uuid: "0ae6b985366f4d47914c40302b102253"
format_stamp: "Formatted at 2026-08-10 23:28:48 on dist-test-slave-rmhg"
server_key: "a60d61e517106eb7f148903bc164cc02"
server_key_iv: "20126f69f44dfc76e4e820faf3c1c8d4"
server_key_version: "encryptionkey@0"
I20260810 23:28:51.288575 6263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:28:51.302081 6263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:28:51.306831 6263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:28:51.307119 6263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:28:51.308324 6263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:28:51.309902 6263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:28:51.310024 6263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:51.310135 6263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:28:51.310209 6263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:51.342844 6263 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.67:37491
I20260810 23:28:51.342921 6469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.67:37491 every 8 connection(s)
I20260810 23:28:51.344436 6263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
I20260810 23:28:51.352468 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 6263
Aug 10 23:28:51 dist-test-slave-rmhg krb5kdc[4807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404531, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:28:51.367877 6054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:51.347270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:38397 (local address 127.14.175.126:42557)
0810 23:28:51.347475 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:51.347480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:51.348787 (+ 1307us) server_negotiation.cc:407] Connection header received
0810 23:28:51.350773 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:51.350779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:51.350870 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:51.350965 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:51.352732 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:51.353900 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:51.356310 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:51.356802 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:51.356906 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:51.361024 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:51.361029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:51.361033 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:51.361074 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:51.364115 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:51.365125 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:51.365130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:51.365133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:51.365324 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:51.365859 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:51.365863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:51.365865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:51.366167 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:51.366310 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:51.367274 (+ 964us) server_negotiation.cc:299] Negotiation successful
0810 23:28:51.367503 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 23:28:51.369123 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:51.347790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.67:38397)
0810 23:28:51.348567 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:51.348625 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:51.350437 (+ 1812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:51.351252 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:51.351262 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:51.351869 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:28:51.352493 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:51.352513 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:51.354147 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:51.354154 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:51.356136 (+ 1982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:51.356154 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:51.357060 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:51.357067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:51.357347 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:51.358050 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:28:51.358083 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:28:51.360790 (+ 2707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:28:51.364278 (+ 3488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:51.364289 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:51.364312 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:51.364999 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:51.365442 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:51.365447 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:51.365451 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:51.365748 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:51.366341 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:28:51.366351 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:28:51.366865 (+ 514us) client_negotiation.cc:769] Sending connection context
0810 23:28:51.367131 (+ 266us) client_negotiation.cc:240] Negotiation successful
0810 23:28:51.367625 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":289,"threads_started":1}
I20260810 23:28:51.369783 6477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:51.353933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59350 (local address 127.14.175.126:42557)
0810 23:28:51.354933 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:51.354937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:51.354951 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:28:51.355001 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:51.355004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:51.355065 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:51.355148 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:51.357048 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:51.357934 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:51.359941 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:51.360421 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:51.360540 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:51.363422 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:51.363429 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:51.363433 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:51.363491 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:51.366685 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:51.367484 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:51.367489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:51.367492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:51.367681 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:51.368260 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:51.368264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:51.368267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:51.368612 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:51.368871 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:51.369132 (+ 261us) server_negotiation.cc:299] Negotiation successful
0810 23:28:51.369404 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":902,"thread_start_us":801,"threads_started":1}
I20260810 23:28:51.370888 6470 heartbeater.cc:347] Connected to a master server at 127.14.175.126:42557
I20260810 23:28:51.371239 6470 heartbeater.cc:464] Registering TS with master...
I20260810 23:28:51.372195 6470 heartbeater.cc:511] Master 127.14.175.126:42557 requested a full tablet report, sending...
I20260810 23:28:51.373726 5743 ts_manager.cc:194] Registered new tserver with Master: 0ae6b985366f4d47914c40302b102253 (127.14.175.67:37491)
I20260810 23:28:51.374615 5743 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.67@KRBTEST.COM'} at 127.14.175.67:38397
I20260810 23:28:51.385010 15037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:28:51.392112 6477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:51.386081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59362 (local address 127.14.175.126:42557)
0810 23:28:51.386285 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:51.386291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:51.386611 (+ 320us) server_negotiation.cc:407] Connection header received
0810 23:28:51.386837 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:51.386841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:51.386988 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:51.387118 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:28:51.387592 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:51.388389 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:51.389909 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:51.390335 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:51.390448 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:51.390982 (+ 534us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:28:51.391539 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:28:51.391678 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:51.391787 (+ 109us) server_negotiation.cc:299] Negotiation successful
0810 23:28:51.391872 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 23:28:51.393114 5743 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:59362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:28:51.400650 5743 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:28:51.413002 5955 tablet_service.cc:1467] Processing CreateTablet for tablet 31d7643768824cdda4325ff4092ce767 (DEFAULT_TABLE table=test-table [id=769e1962b253470a8de5b7aed3f96130]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:28:51.414084 5955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31d7643768824cdda4325ff4092ce767. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:28:51.425112 6489 tablet_bootstrap.cc:492] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf: Bootstrap starting.
I20260810 23:28:51.427089 6489 tablet_bootstrap.cc:654] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf: Neither blocks nor log segments found. Creating new log.
I20260810 23:28:51.429247 6489 tablet_bootstrap.cc:492] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf: No bootstrap required, opened a new log
I20260810 23:28:51.429414 6489 ts_tablet_manager.cc:1397] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20260810 23:28:51.430261 6489 raft_consensus.cc:348] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030a6b9f073b4e8b827b8d8d166cf958" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44945 } } peers { permanent_uuid: "0ae6b985366f4d47914c40302b102253" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37491 } } peers { permanent_uuid: "c2025f4fdf524504bfe241da57e32cbf" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35659 } }
I20260810 23:28:51.430529 6489 raft_consensus.cc:374] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:28:51.430641 6489 raft_consensus.cc:729] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2025f4fdf524504bfe241da57e32cbf, State: Initialized, Role: FOLLOWER
I20260810 23:28:51.430922 6489 consensus_queue.cc:260] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030a6b9f073b4e8b827b8d8d166cf958" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44945 } } peers { permanent_uuid: "0ae6b985366f4d47914c40302b102253" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37491 } } peers { permanent_uuid: "c2025f4fdf524504bfe241da57e32cbf" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35659 } }
I20260810 23:28:51.434986 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:51.411437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45716 (local address 127.14.175.66:44945)
0810 23:28:51.422545 (+ 11108us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:51.422554 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:51.425934 (+ 3380us) server_negotiation.cc:407] Connection header received
0810 23:28:51.426333 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:51.426344 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:51.426652 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:51.426854 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:28:51.428112 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:51.429996 (+ 1884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:51.432833 (+ 2837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:51.433881 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:51.434012 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:51.434229 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:51.434499 (+ 270us) server_negotiation.cc:299] Negotiation successful
0810 23:28:51.434683 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10958,"thread_start_us":10835,"threads_started":1}
I20260810 23:28:51.435006 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:51.411137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:44945 (local address 127.0.0.1:45716)
0810 23:28:51.425779 (+ 14642us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:51.425808 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:51.426079 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:51.427417 (+ 1338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:51.427423 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:51.427452 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:28:51.427872 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:51.427887 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:51.430154 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:51.430160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:51.432495 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:51.432518 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:51.434096 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:51.434103 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:51.434305 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:51.434337 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 23:28:51.434422 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 23:28:51.434566 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14491,"mutex_wait_us":1,"thread_start_us":15453,"threads_started":1}
I20260810 23:28:51.437166 6487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:51.411936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51012 (local address 127.14.175.67:37491)
0810 23:28:51.422239 (+ 10303us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:51.422251 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:51.426976 (+ 4725us) server_negotiation.cc:407] Connection header received
0810 23:28:51.427114 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:51.427120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:51.427451 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:51.427616 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:28:51.428280 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:51.430071 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:51.432715 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:51.434358 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:51.434498 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:51.434742 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:51.436565 (+ 1823us) server_negotiation.cc:299] Negotiation successful
0810 23:28:51.436778 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10061,"thread_start_us":9873,"threads_started":1}
I20260810 23:28:51.437399 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:51.411593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:37491 (local address 127.0.0.1:51012)
0810 23:28:51.426832 (+ 15239us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:51.426849 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:51.426974 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:51.427634 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:51.427639 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:51.427663 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:28:51.428069 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:51.428083 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:51.430233 (+ 2150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:51.430237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:51.432548 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:51.432558 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:51.434930 (+ 2372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:51.434940 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:51.435694 (+ 754us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:51.435740 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 23:28:51.435840 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 23:28:51.435998 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15122,"mutex_wait_us":26,"thread_start_us":13875,"threads_started":1}
I20260810 23:28:51.438673 6489 ts_tablet_manager.cc:1428] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf: Time spent starting tablet: real 0.009s user 0.005s sys 0.003s
I20260810 23:28:51.438961 6154 tablet_service.cc:1467] Processing CreateTablet for tablet 31d7643768824cdda4325ff4092ce767 (DEFAULT_TABLE table=test-table [id=769e1962b253470a8de5b7aed3f96130]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:28:51.440459 6154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31d7643768824cdda4325ff4092ce767. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:28:51.440963 6356 tablet_service.cc:1467] Processing CreateTablet for tablet 31d7643768824cdda4325ff4092ce767 (DEFAULT_TABLE table=test-table [id=769e1962b253470a8de5b7aed3f96130]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:28:51.442345 6356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31d7643768824cdda4325ff4092ce767. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:28:51.454326 6494 tablet_bootstrap.cc:492] T 31d7643768824cdda4325ff4092ce767 P 030a6b9f073b4e8b827b8d8d166cf958: Bootstrap starting.
I20260810 23:28:51.455231 6498 tablet_bootstrap.cc:492] T 31d7643768824cdda4325ff4092ce767 P 0ae6b985366f4d47914c40302b102253: Bootstrap starting.
I20260810 23:28:51.457710 6494 tablet_bootstrap.cc:654] T 31d7643768824cdda4325ff4092ce767 P 030a6b9f073b4e8b827b8d8d166cf958: Neither blocks nor log segments found. Creating new log.
I20260810 23:28:51.458231 6498 tablet_bootstrap.cc:654] T 31d7643768824cdda4325ff4092ce767 P 0ae6b985366f4d47914c40302b102253: Neither blocks nor log segments found. Creating new log.
I20260810 23:28:51.458807 6494 log.cc:824] T 31d7643768824cdda4325ff4092ce767 P 030a6b9f073b4e8b827b8d8d166cf958: Log is configured to *not* fsync() on all Append() calls
I20260810 23:28:51.459295 6498 log.cc:824] T 31d7643768824cdda4325ff4092ce767 P 0ae6b985366f4d47914c40302b102253: Log is configured to *not* fsync() on all Append() calls
I20260810 23:28:51.462007 6494 tablet_bootstrap.cc:492] T 31d7643768824cdda4325ff4092ce767 P 030a6b9f073b4e8b827b8d8d166cf958: No bootstrap required, opened a new log
I20260810 23:28:51.462288 6494 ts_tablet_manager.cc:1397] T 31d7643768824cdda4325ff4092ce767 P 030a6b9f073b4e8b827b8d8d166cf958: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 23:28:51.462827 6498 tablet_bootstrap.cc:492] T 31d7643768824cdda4325ff4092ce767 P 0ae6b985366f4d47914c40302b102253: No bootstrap required, opened a new log
I20260810 23:28:51.463057 6498 ts_tablet_manager.cc:1397] T 31d7643768824cdda4325ff4092ce767 P 0ae6b985366f4d47914c40302b102253: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 23:28:51.469578 6494 raft_consensus.cc:348] T 31d7643768824cdda4325ff4092ce767 P 030a6b9f073b4e8b827b8d8d166cf958 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030a6b9f073b4e8b827b8d8d166cf958" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44945 } } peers { permanent_uuid: "0ae6b985366f4d47914c40302b102253" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37491 } } peers { permanent_uuid: "c2025f4fdf524504bfe241da57e32cbf" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35659 } }
I20260810 23:28:51.470034 6494 raft_consensus.cc:374] T 31d7643768824cdda4325ff4092ce767 P 030a6b9f073b4e8b827b8d8d166cf958 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:28:51.470131 6494 raft_consensus.cc:729] T 31d7643768824cdda4325ff4092ce767 P 030a6b9f073b4e8b827b8d8d166cf958 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 030a6b9f073b4e8b827b8d8d166cf958, State: Initialized, Role: FOLLOWER
I20260810 23:28:51.470278 6498 raft_consensus.cc:348] T 31d7643768824cdda4325ff4092ce767 P 0ae6b985366f4d47914c40302b102253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030a6b9f073b4e8b827b8d8d166cf958" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44945 } } peers { permanent_uuid: "0ae6b985366f4d47914c40302b102253" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37491 } } peers { permanent_uuid: "c2025f4fdf524504bfe241da57e32cbf" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35659 } }
I20260810 23:28:51.470710 6498 raft_consensus.cc:374] T 31d7643768824cdda4325ff4092ce767 P 0ae6b985366f4d47914c40302b102253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:28:51.470809 6498 raft_consensus.cc:729] T 31d7643768824cdda4325ff4092ce767 P 0ae6b985366f4d47914c40302b102253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ae6b985366f4d47914c40302b102253, State: Initialized, Role: FOLLOWER
I20260810 23:28:51.470700 6494 consensus_queue.cc:260] T 31d7643768824cdda4325ff4092ce767 P 030a6b9f073b4e8b827b8d8d166cf958 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030a6b9f073b4e8b827b8d8d166cf958" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44945 } } peers { permanent_uuid: "0ae6b985366f4d47914c40302b102253" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37491 } } peers { permanent_uuid: "c2025f4fdf524504bfe241da57e32cbf" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35659 } }
I20260810 23:28:51.471380 6498 consensus_queue.cc:260] T 31d7643768824cdda4325ff4092ce767 P 0ae6b985366f4d47914c40302b102253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030a6b9f073b4e8b827b8d8d166cf958" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44945 } } peers { permanent_uuid: "0ae6b985366f4d47914c40302b102253" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37491 } } peers { permanent_uuid: "c2025f4fdf524504bfe241da57e32cbf" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35659 } }
I20260810 23:28:51.471926 6256 heartbeater.cc:503] Master 127.14.175.126:42557 was elected leader, sending a full tablet report...
I20260810 23:28:51.472460 6470 heartbeater.cc:503] Master 127.14.175.126:42557 was elected leader, sending a full tablet report...
I20260810 23:28:51.472620 6494 ts_tablet_manager.cc:1428] T 31d7643768824cdda4325ff4092ce767 P 030a6b9f073b4e8b827b8d8d166cf958: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260810 23:28:51.473101 6498 ts_tablet_manager.cc:1428] T 31d7643768824cdda4325ff4092ce767 P 0ae6b985366f4d47914c40302b102253: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260810 23:28:51.539760 6493 raft_consensus.cc:482] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:28:51.540112 6493 raft_consensus.cc:504] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030a6b9f073b4e8b827b8d8d166cf958" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44945 } } peers { permanent_uuid: "0ae6b985366f4d47914c40302b102253" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37491 } } peers { permanent_uuid: "c2025f4fdf524504bfe241da57e32cbf" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35659 } }
I20260810 23:28:51.541803 6493 leader_election.cc:288] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 030a6b9f073b4e8b827b8d8d166cf958 (127.14.175.66:44945), 0ae6b985366f4d47914c40302b102253 (127.14.175.67:37491)
I20260810 23:28:51.550534 6487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:51.542288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:52015 (local address 127.14.175.67:37491)
0810 23:28:51.542504 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:51.542511 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:51.543165 (+ 654us) server_negotiation.cc:407] Connection header received
0810 23:28:51.543366 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:51.543371 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:51.543453 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:51.543553 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:28:51.544292 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:51.545477 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:51.548269 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:51.549525 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:51.549662 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:51.549736 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:51.550132 (+ 396us) server_negotiation.cc:299] Negotiation successful
0810 23:28:51.550267 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 23:28:51.550536 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:51.542288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:39137 (local address 127.14.175.66:44945)
0810 23:28:51.542504 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:51.542511 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:51.543247 (+ 736us) server_negotiation.cc:407] Connection header received
0810 23:28:51.543365 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:51.543371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:51.543454 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:51.543553 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:28:51.544363 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:51.545477 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:51.548262 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:51.549488 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:51.549653 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:51.549732 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:51.550132 (+ 400us) server_negotiation.cc:299] Negotiation successful
0810 23:28:51.550267 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 23:28:51.550879 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:51.542172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:37491 (local address 127.14.175.65:52015)
0810 23:28:51.543025 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:51.543046 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:51.543216 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:51.543581 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:51.543586 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:51.543630 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:28:51.544103 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:51.544123 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:51.545601 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:51.545606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:51.548090 (+ 2484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:51.548104 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:51.549697 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:51.549702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:51.549934 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:51.550004 (+ 70us) client_negotiation.cc:769] Sending connection context
0810 23:28:51.550106 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 23:28:51.550298 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":221,"spinlock_wait_cycles":4992,"thread_start_us":516,"threads_started":1}
I20260810 23:28:51.551080 6509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:51.542199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:44945 (local address 127.14.175.65:39137)
0810 23:28:51.543134 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:51.543159 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:51.543256 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:51.543581 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:51.543586 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:51.543623 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:28:51.544186 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:51.544197 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:51.545604 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:51.545608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:51.548087 (+ 2479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:51.548104 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:51.549668 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:51.549676 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:51.549925 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:51.550004 (+ 79us) client_negotiation.cc:769] Sending connection context
0810 23:28:51.550106 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 23:28:51.550298 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"mutex_wait_us":233,"spinlock_wait_cycles":1408,"thread_start_us":600,"threads_started":1}
I20260810 23:28:51.552014 6391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31d7643768824cdda4325ff4092ce767" candidate_uuid: "c2025f4fdf524504bfe241da57e32cbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ae6b985366f4d47914c40302b102253" is_pre_election: true
I20260810 23:28:51.552065 6184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31d7643768824cdda4325ff4092ce767" candidate_uuid: "c2025f4fdf524504bfe241da57e32cbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "030a6b9f073b4e8b827b8d8d166cf958" is_pre_election: true
I20260810 23:28:51.552526 6184 raft_consensus.cc:2454] T 31d7643768824cdda4325ff4092ce767 P 030a6b9f073b4e8b827b8d8d166cf958 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2025f4fdf524504bfe241da57e32cbf in term 0.
I20260810 23:28:51.552526 6391 raft_consensus.cc:2454] T 31d7643768824cdda4325ff4092ce767 P 0ae6b985366f4d47914c40302b102253 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2025f4fdf524504bfe241da57e32cbf in term 0.
I20260810 23:28:51.553282 5881 leader_election.cc:302] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ae6b985366f4d47914c40302b102253, c2025f4fdf524504bfe241da57e32cbf; no voters:
I20260810 23:28:51.553785 6493 raft_consensus.cc:2781] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:28:51.553956 6493 raft_consensus.cc:482] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:28:51.554060 6493 raft_consensus.cc:3037] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:28:51.556360 6493 raft_consensus.cc:504] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030a6b9f073b4e8b827b8d8d166cf958" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44945 } } peers { permanent_uuid: "0ae6b985366f4d47914c40302b102253" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37491 } } peers { permanent_uuid: "c2025f4fdf524504bfe241da57e32cbf" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35659 } }
I20260810 23:28:51.556943 6493 leader_election.cc:288] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [CANDIDATE]: Term 1 election: Requested vote from peers 030a6b9f073b4e8b827b8d8d166cf958 (127.14.175.66:44945), 0ae6b985366f4d47914c40302b102253 (127.14.175.67:37491)
I20260810 23:28:51.557477 6391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31d7643768824cdda4325ff4092ce767" candidate_uuid: "c2025f4fdf524504bfe241da57e32cbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ae6b985366f4d47914c40302b102253"
I20260810 23:28:51.557480 6184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31d7643768824cdda4325ff4092ce767" candidate_uuid: "c2025f4fdf524504bfe241da57e32cbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "030a6b9f073b4e8b827b8d8d166cf958"
I20260810 23:28:51.557663 6391 raft_consensus.cc:3037] T 31d7643768824cdda4325ff4092ce767 P 0ae6b985366f4d47914c40302b102253 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:28:51.557684 6184 raft_consensus.cc:3037] T 31d7643768824cdda4325ff4092ce767 P 030a6b9f073b4e8b827b8d8d166cf958 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:28:51.559968 6184 raft_consensus.cc:2454] T 31d7643768824cdda4325ff4092ce767 P 030a6b9f073b4e8b827b8d8d166cf958 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2025f4fdf524504bfe241da57e32cbf in term 1.
I20260810 23:28:51.560017 6391 raft_consensus.cc:2454] T 31d7643768824cdda4325ff4092ce767 P 0ae6b985366f4d47914c40302b102253 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2025f4fdf524504bfe241da57e32cbf in term 1.
I20260810 23:28:51.560432 5875 leader_election.cc:302] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 030a6b9f073b4e8b827b8d8d166cf958, c2025f4fdf524504bfe241da57e32cbf; no voters:
I20260810 23:28:51.560772 6493 raft_consensus.cc:2781] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:28:51.561268 6493 raft_consensus.cc:686] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [term 1 LEADER]: Becoming Leader. State: Replica: c2025f4fdf524504bfe241da57e32cbf, State: Running, Role: LEADER
I20260810 23:28:51.561647 6493 consensus_queue.cc:237] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030a6b9f073b4e8b827b8d8d166cf958" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44945 } } peers { permanent_uuid: "0ae6b985366f4d47914c40302b102253" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37491 } } peers { permanent_uuid: "c2025f4fdf524504bfe241da57e32cbf" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35659 } }
I20260810 23:28:51.566021 5750 catalog_manager.cc:5495] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf reported cstate change: term changed from 0 to 1, leader changed from <none> to c2025f4fdf524504bfe241da57e32cbf (127.14.175.65). New cstate: current_term: 1 leader_uuid: "c2025f4fdf524504bfe241da57e32cbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030a6b9f073b4e8b827b8d8d166cf958" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ae6b985366f4d47914c40302b102253" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2025f4fdf524504bfe241da57e32cbf" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35659 } health_report { overall_health: HEALTHY } } }
W20260810 23:28:51.596118 6472 tablet.cc:2369] T 31d7643768824cdda4325ff4092ce767 P 0ae6b985366f4d47914c40302b102253: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:28:51.597626 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:51.591309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40106 (local address 127.14.175.65:35659)
0810 23:28:51.591966 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:51.591972 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:51.592002 (+ 30us) server_negotiation.cc:407] Connection header received
0810 23:28:51.592107 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:51.592111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:51.592201 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:51.592356 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:28:51.592842 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:51.593727 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:51.595217 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:51.595650 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:51.595772 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:51.596276 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:28:51.596923 (+ 647us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:28:51.597063 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:51.597212 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 23:28:51.597300 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":353,"threads_started":1}
I20260810 23:28:51.603933 6391 raft_consensus.cc:1261] T 31d7643768824cdda4325ff4092ce767 P 0ae6b985366f4d47914c40302b102253 [term 1 FOLLOWER]: Refusing update from remote peer c2025f4fdf524504bfe241da57e32cbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:28:51.603961 6184 raft_consensus.cc:1261] T 31d7643768824cdda4325ff4092ce767 P 030a6b9f073b4e8b827b8d8d166cf958 [term 1 FOLLOWER]: Refusing update from remote peer c2025f4fdf524504bfe241da57e32cbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:28:51.605043 6510 consensus_queue.cc:1035] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ae6b985366f4d47914c40302b102253" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:28:51.605474 6493 consensus_queue.cc:1035] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "030a6b9f073b4e8b827b8d8d166cf958" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:28:51.617772 6517 mvcc.cc:205] Tried to move back new op lower bound from 7317112961440428032 to 7317112961280274432. Current Snapshot: MvccSnapshot[applied={T|T < 7317112961440428032}]
I20260810 23:28:51.619374 6519 mvcc.cc:205] Tried to move back new op lower bound from 7317112961440428032 to 7317112961280274432. Current Snapshot: MvccSnapshot[applied={T|T < 7317112961440428032}]
W20260810 23:28:51.623469 6259 tablet.cc:2369] T 31d7643768824cdda4325ff4092ce767 P 030a6b9f073b4e8b827b8d8d166cf958: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:28:51.643163 5743 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:59362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:28:51.643435 5743 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:59362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:28:51.647424 5743 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102: Sending DeleteTablet for 3 replicas of tablet 31d7643768824cdda4325ff4092ce767
I20260810 23:28:51.648698 6154 tablet_service.cc:1514] Processing DeleteTablet for tablet 31d7643768824cdda4325ff4092ce767 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:28:51 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:45716
I20260810 23:28:51.649119 6356 tablet_service.cc:1514] Processing DeleteTablet for tablet 31d7643768824cdda4325ff4092ce767 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:28:51 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:51012
I20260810 23:28:51.649418 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 5816
I20260810 23:28:51.650463 6535 tablet_replica.cc:330] T 31d7643768824cdda4325ff4092ce767 P 030a6b9f073b4e8b827b8d8d166cf958: stopping tablet replica
I20260810 23:28:51.650985 6535 raft_consensus.cc:2229] T 31d7643768824cdda4325ff4092ce767 P 030a6b9f073b4e8b827b8d8d166cf958 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:28:51.651245 5955 tablet_service.cc:1514] Processing DeleteTablet for tablet 31d7643768824cdda4325ff4092ce767 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:28:51 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:40104
I20260810 23:28:51.651552 6535 raft_consensus.cc:2258] T 31d7643768824cdda4325ff4092ce767 P 030a6b9f073b4e8b827b8d8d166cf958 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:28:51.651613 6536 tablet_replica.cc:330] T 31d7643768824cdda4325ff4092ce767 P 0ae6b985366f4d47914c40302b102253: stopping tablet replica
I20260810 23:28:51.652133 6536 raft_consensus.cc:2229] T 31d7643768824cdda4325ff4092ce767 P 0ae6b985366f4d47914c40302b102253 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:28:51.654035 6535 ts_tablet_manager.cc:1905] T 31d7643768824cdda4325ff4092ce767 P 030a6b9f073b4e8b827b8d8d166cf958: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:28:51.654698 6536 raft_consensus.cc:2258] T 31d7643768824cdda4325ff4092ce767 P 0ae6b985366f4d47914c40302b102253 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:28:51.655588 6538 tablet_replica.cc:330] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf: stopping tablet replica
I20260810 23:28:51.659348 6536 ts_tablet_manager.cc:1905] T 31d7643768824cdda4325ff4092ce767 P 0ae6b985366f4d47914c40302b102253: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:28:51.661461 6535 ts_tablet_manager.cc:1918] T 31d7643768824cdda4325ff4092ce767 P 030a6b9f073b4e8b827b8d8d166cf958: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:28:51.661667 6535 log.cc:1196] T 31d7643768824cdda4325ff4092ce767 P 030a6b9f073b4e8b827b8d8d166cf958: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/wal/wals/31d7643768824cdda4325ff4092ce767
I20260810 23:28:51.662314 6535 ts_tablet_manager.cc:1939] T 31d7643768824cdda4325ff4092ce767 P 030a6b9f073b4e8b827b8d8d166cf958: Deleting consensus metadata
I20260810 23:28:51.664354 5722 catalog_manager.cc:4841] TS 030a6b9f073b4e8b827b8d8d166cf958 (127.14.175.66:44945): tablet 31d7643768824cdda4325ff4092ce767 (table test-table [id=769e1962b253470a8de5b7aed3f96130]) successfully deleted
I20260810 23:28:51.665750 6538 raft_consensus.cc:2229] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:28:51.666751 6536 ts_tablet_manager.cc:1918] T 31d7643768824cdda4325ff4092ce767 P 0ae6b985366f4d47914c40302b102253: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:28:51.666946 6536 log.cc:1196] T 31d7643768824cdda4325ff4092ce767 P 0ae6b985366f4d47914c40302b102253: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/wal/wals/31d7643768824cdda4325ff4092ce767
I20260810 23:28:51.667557 6536 ts_tablet_manager.cc:1939] T 31d7643768824cdda4325ff4092ce767 P 0ae6b985366f4d47914c40302b102253: Deleting consensus metadata
I20260810 23:28:51.668174 6538 raft_consensus.cc:2258] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:28:51.669642 5725 catalog_manager.cc:4841] TS 0ae6b985366f4d47914c40302b102253 (127.14.175.67:37491): tablet 31d7643768824cdda4325ff4092ce767 (table test-table [id=769e1962b253470a8de5b7aed3f96130]) successfully deleted
I20260810 23:28:51.670568 6538 ts_tablet_manager.cc:1905] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:28:51.671149 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:51.652173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40118 (local address 127.14.175.65:35659)
0810 23:28:51.652360 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:51.652366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:51.652386 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:28:51.652452 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:51.652457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:51.652541 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:51.652668 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:51.654516 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:51.655708 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:51.657795 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:51.658500 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:51.658635 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:51.663365 (+ 4730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:51.663384 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:51.663403 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:51.663452 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:51.667399 (+ 3947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:51.668491 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:51.668499 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:51.668502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:51.668660 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:51.669139 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:51.669209 (+ 70us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:51.669213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:51.669682 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:51.669986 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:51.670395 (+ 409us) server_negotiation.cc:299] Negotiation successful
0810 23:28:51.670659 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 23:28:51.671834 6039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:28:51.774098 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 5816
W20260810 23:28:51.801716 5723 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.175.65:35659 (error 108)
I20260810 23:28:51.801920 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 6057
I20260810 23:28:51.803915 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:51.803391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:35659 (local address 127.0.0.1:40134)
0810 23:28:51.803595 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:51.803738 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260810 23:28:51.804224 5723 catalog_manager.cc:4584] TS c2025f4fdf524504bfe241da57e32cbf (127.14.175.65:35659): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 31d7643768824cdda4325ff4092ce767: Network error: Client connection negotiation failed: client connection to 127.14.175.65:35659: connect: Connection refused (error 111)
I20260810 23:28:51.819938 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:51.802768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45732 (local address 127.14.175.66:44945)
0810 23:28:51.802935 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:51.802940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:51.803055 (+ 115us) server_negotiation.cc:407] Connection header received
0810 23:28:51.803282 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:51.803288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:51.803375 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:51.803488 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:51.805572 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:51.806462 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:51.807836 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:51.808266 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:51.808371 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:51.814166 (+ 5795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:51.814185 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:51.814203 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:51.814247 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:51.817238 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:51.818004 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:51.818014 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:51.818017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:51.818159 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:51.818459 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:51.818462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:51.818464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:51.818920 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:51.819164 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:51.819430 (+ 266us) server_negotiation.cc:299] Negotiation successful
0810 23:28:51.819668 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:28:51.820716 6248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:28:51.857954 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:51.857371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:35659 (local address 127.0.0.1:40144)
0810 23:28:51.857612 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:51.857756 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 23:28:51.859105 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:51.858690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:35659 (local address 127.0.0.1:40148)
0810 23:28:51.858859 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:51.858964 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 23:28:51.919785 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:51.919199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:35659 (local address 127.0.0.1:40150)
0810 23:28:51.919460 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:51.919581 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260810 23:28:51.920946 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:51.920498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:35659 (local address 127.0.0.1:40164)
0810 23:28:51.920715 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:51.920802 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260810 23:28:51.921656 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 6057
I20260810 23:28:51.947752 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 6263
I20260810 23:28:51.960816 6487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:51.948514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51026 (local address 127.14.175.67:37491)
0810 23:28:51.948677 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:51.948681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:51.948742 (+ 61us) server_negotiation.cc:407] Connection header received
0810 23:28:51.948911 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:51.948915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:51.948993 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:51.949071 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:51.950468 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:51.951211 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:51.952528 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:51.952910 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:51.952998 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:51.955564 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:51.955580 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:51.955600 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:51.955637 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:51.958251 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:51.958993 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:51.959000 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:51.959002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:51.959166 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:51.959544 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:51.959547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:51.959549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:51.959912 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:51.960141 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:51.960393 (+ 252us) server_negotiation.cc:299] Negotiation successful
0810 23:28:51.960572 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:28:51.961615 6462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:28:52.002012 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:52.001399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:35659 (local address 127.0.0.1:40170)
0810 23:28:52.001648 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:52.001783 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260810 23:28:52.003219 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:52.002742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:35659 (local address 127.0.0.1:40176)
0810 23:28:52.002975 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:52.003073 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260810 23:28:52.049968 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 6263
I20260810 23:28:52.071820 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 5693
I20260810 23:28:52.073206 5805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:28:52.190565 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:52.189783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:35659 (local address 127.0.0.1:40182)
0810 23:28:52.190169 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:52.190295 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223}
I20260810 23:28:52.190824 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 5693
I20260810 23:28:52.219478 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:42557
--webserver_interface=127.14.175.126
--webserver_port=45543
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:42173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:42557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:28:52.565891 6549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:28:52.566203 6549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:28:52.566334 6549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:28:52.573710 6549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:28:52.573840 6549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:28:52.573900 6549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:28:52.573980 6549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:28:52.574035 6549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:28:52.584438 6549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:42173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:42557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:42557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=45543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:28:52.585999 6549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:28:52.587423 6549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:28:52.600011 6557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:52.599977 6562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:52.599948 6559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:28:52.600760 6549 server_base.cc:1034] running on GCE node
I20260810 23:28:52.601678 6549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:28:52.603312 6549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:28:52.604542 6549 hybrid_clock.cc:648] HybridClock initialized: now 1786404532604478 us; error 60 us; skew 500 ppm
I20260810 23:28:52.604926 6549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:28:52 dist-test-slave-rmhg krb5kdc[4807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404532, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:28:52.608197 6549 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:28:52.609789 6549 webserver.cc:460] Webserver started at http://127.14.175.126:45543/ using document root <none> and password file <none>
I20260810 23:28:52.610518 6549 fs_manager.cc:359] Metadata directory not provided
I20260810 23:28:52.610630 6549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:28:52.616446 6549 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:28:52.619421 6574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:52.620839 6549 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:28:52.621006 6549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "eb2dc92dfa794feb958788b902dff102"
format_stamp: "Formatted at 2026-08-10 23:28:46 on dist-test-slave-rmhg"
server_key: "1bd56fd22eb3dbba1fed953350aff19f"
server_key_iv: "eff27a950289987b7c118f5a7d751645"
server_key_version: "encryptionkey@0"
I20260810 23:28:52.622032 6549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:28:52.673980 6549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:28:52.678488 6549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:28:52.678756 6549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:28:52.698133 6549 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:42557
I20260810 23:28:52.698185 6660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:42557 every 8 connection(s)
I20260810 23:28:52.699776 6549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:28:52.707610 6661 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:28:52.708899 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 6549
I20260810 23:28:52.709786 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.65:35659
--local_ip_for_outbound_sockets=127.14.175.65
--tserver_master_addrs=127.14.175.126:42557
--webserver_port=33167
--webserver_interface=127.14.175.65
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:42173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:28:52.711355 6661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102: Bootstrap starting.
I20260810 23:28:52.733860 6661 log.cc:824] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102: Log is configured to *not* fsync() on all Append() calls
I20260810 23:28:52.751987 6661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:28:52.752600 6661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102: Bootstrap complete.
I20260810 23:28:52.762238 6661 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2dc92dfa794feb958788b902dff102" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 42557 } }
I20260810 23:28:52.763199 6661 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2dc92dfa794feb958788b902dff102, State: Initialized, Role: FOLLOWER
I20260810 23:28:52.763854 6661 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2dc92dfa794feb958788b902dff102" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 42557 } }
I20260810 23:28:52.764118 6661 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:28:52.764243 6661 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:28:52.764408 6661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [term 2 FOLLOWER]: Advancing to term 3
I20260810 23:28:52.768564 6661 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2dc92dfa794feb958788b902dff102" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 42557 } }
I20260810 23:28:52.769199 6661 leader_election.cc:302] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb2dc92dfa794feb958788b902dff102; no voters:
I20260810 23:28:52.769901 6661 leader_election.cc:288] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 23:28:52.770057 6666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 23:28:52.771356 6666 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [term 3 LEADER]: Becoming Leader. State: Replica: eb2dc92dfa794feb958788b902dff102, State: Running, Role: LEADER
I20260810 23:28:52.772013 6666 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2dc92dfa794feb958788b902dff102" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 42557 } }
I20260810 23:28:52.772325 6661 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:28:52.774778 6674 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb2dc92dfa794feb958788b902dff102. Latest consensus state: current_term: 3 leader_uuid: "eb2dc92dfa794feb958788b902dff102" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2dc92dfa794feb958788b902dff102" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 42557 } } }
I20260810 23:28:52.775213 6674 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [sys.catalog]: This master's current role is: LEADER
I20260810 23:28:52.775684 6667 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "eb2dc92dfa794feb958788b902dff102" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2dc92dfa794feb958788b902dff102" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 42557 } } }
I20260810 23:28:52.775995 6667 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb2dc92dfa794feb958788b902dff102 [sys.catalog]: This master's current role is: LEADER
I20260810 23:28:52.776247 6682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:28:52.798482 6682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:28:52.800097 6682 catalog_manager.cc:1259] Loaded cluster ID: faaa53eb7bf34fd7ad248f8c6b08f82d
I20260810 23:28:52.800300 6682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 23:28:52.802752 6682 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6bbabd69ad google::LogMessage::Fail() at ??:0
@ 0x7f6bbabd988a google::LogMessage::SendToLog() at ??:0
@ 0x7f6bbabd64de google::LogMessage::Flush() at ??:0
@ 0x7f6bbabd7a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6bd060d512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f6bd060a976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6bd0692610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f6bd0692439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f6bbd822345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6bbd82591c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f6bbd825799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f6bbd800eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6bc0ada6db start_thread at ??:0
@ 0x7f6bb9a7061f clone at ??:0
W20260810 23:28:53.167711 6663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:28:53.168092 6663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:28:53.168210 6663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:28:53.175923 6663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:28:53.176069 6663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:28:53.176200 6663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.65
I20260810 23:28:53.186990 6663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:42173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.65:35659
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.175.65
--webserver_port=33167
--tserver_master_addrs=127.14.175.126:42557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.65
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:28:53.188369 6663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:28:53.189823 6663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:28:53.203935 6737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:53.203909 6738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:53.203991 6742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:28:53.204746 6663 server_base.cc:1034] running on GCE node
I20260810 23:28:53.205482 6663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:28:53.206573 6663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:28:53.207837 6663 hybrid_clock.cc:648] HybridClock initialized: now 1786404533207775 us; error 45 us; skew 500 ppm
I20260810 23:28:53.208206 6663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:28:53 dist-test-slave-rmhg krb5kdc[4807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404533, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:28:53.211755 6663 init.cc:377] Logged in from keytab as kudu/127.14.175.65@KRBTEST.COM (short username kudu)
I20260810 23:28:53.213555 6663 webserver.cc:460] Webserver started at http://127.14.175.65:33167/ using document root <none> and password file <none>
I20260810 23:28:53.214299 6663 fs_manager.cc:359] Metadata directory not provided
I20260810 23:28:53.214418 6663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:28:53.220583 6663 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:28:53.224244 6755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:53.226022 6663 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:28:53.226199 6663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/wal
uuid: "c2025f4fdf524504bfe241da57e32cbf"
format_stamp: "Formatted at 2026-08-10 23:28:47 on dist-test-slave-rmhg"
server_key: "d49574e7b52e2633edc868116aa20f0c"
server_key_iv: "5c53a2d3a339317e7c4c5fd990c289b6"
server_key_version: "encryptionkey@0"
I20260810 23:28:53.227272 6663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:28:53.241786 6663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:28:53.246587 6663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:28:53.246887 6663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:28:53.247951 6663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:28:53.249913 6766 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260810 23:28:53.256031 6663 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 23:28:53.256191 6663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260810 23:28:53.256326 6663 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 23:28:53.259732 6663 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 23:28:53.259822 6663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.001s sys 0.000s
I20260810 23:28:53.260232 6766 tablet_bootstrap.cc:492] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf: Bootstrap starting.
I20260810 23:28:53.269793 6766 log.cc:824] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf: Log is configured to *not* fsync() on all Append() calls
I20260810 23:28:53.284945 6766 tablet_bootstrap.cc:492] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:28:53.285974 6766 tablet_bootstrap.cc:492] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf: Bootstrap complete.
I20260810 23:28:53.286753 6766 ts_tablet_manager.cc:1397] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.012s
I20260810 23:28:53.296077 6766 raft_consensus.cc:348] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030a6b9f073b4e8b827b8d8d166cf958" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44945 } } peers { permanent_uuid: "0ae6b985366f4d47914c40302b102253" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37491 } } peers { permanent_uuid: "c2025f4fdf524504bfe241da57e32cbf" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35659 } }
I20260810 23:28:53.297557 6766 raft_consensus.cc:729] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2025f4fdf524504bfe241da57e32cbf, State: Initialized, Role: FOLLOWER
I20260810 23:28:53.298452 6766 consensus_queue.cc:260] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030a6b9f073b4e8b827b8d8d166cf958" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44945 } } peers { permanent_uuid: "0ae6b985366f4d47914c40302b102253" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37491 } } peers { permanent_uuid: "c2025f4fdf524504bfe241da57e32cbf" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35659 } }
I20260810 23:28:53.301067 6766 ts_tablet_manager.cc:1428] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260810 23:28:53.309955 6663 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.65:35659
I20260810 23:28:53.309999 6938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.65:35659 every 8 connection(s)
I20260810 23:28:53.311412 6663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
I20260810 23:28:53.316613 6943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:53.314612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.65:49633)
0810 23:28:53.315442 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:53.315700 (+ 258us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":327,"threads_started":1}
W20260810 23:28:53.317381 6939 heartbeater.cc:644] Failed to heartbeat to 127.14.175.126:42557 (0 consecutive failures): Network error: Failed to ping master at 127.14.175.126:42557: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
I20260810 23:28:53.318310 6943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:53.317975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.65:57777)
0810 23:28:53.318124 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:53.318200 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 23:28:53.319162 6943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:53.318844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.65:41683)
0810 23:28:53.318966 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:53.319040 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 23:28:53.319396 6939 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 23:28:53.321143 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 6663
I20260810 23:28:53.321830 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.66:44945
--local_ip_for_outbound_sockets=127.14.175.66
--tserver_master_addrs=127.14.175.126:42557
--webserver_port=41887
--webserver_interface=127.14.175.66
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:42173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:28:53.673408 6944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:28:53.673743 6944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:28:53.673874 6944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:28:53.681422 6944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:28:53.681556 6944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:28:53.681711 6944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.66
I20260810 23:28:53.692018 6944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:42173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.66:44945
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.175.66
--webserver_port=41887
--tserver_master_addrs=127.14.175.126:42557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.66
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:28:53.693499 6944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:28:53.694957 6944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:28:53.709955 6952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:53.709988 6954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:53.710011 6961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:28:53.710834 6944 server_base.cc:1034] running on GCE node
I20260810 23:28:53.711535 6944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:28:53.712598 6944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:28:53.713850 6944 hybrid_clock.cc:648] HybridClock initialized: now 1786404533713756 us; error 80 us; skew 500 ppm
I20260810 23:28:53.714305 6944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:28:53 dist-test-slave-rmhg krb5kdc[4807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404533, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:28:53.717890 6944 init.cc:377] Logged in from keytab as kudu/127.14.175.66@KRBTEST.COM (short username kudu)
I20260810 23:28:53.719604 6944 webserver.cc:460] Webserver started at http://127.14.175.66:41887/ using document root <none> and password file <none>
I20260810 23:28:53.720394 6944 fs_manager.cc:359] Metadata directory not provided
I20260810 23:28:53.720515 6944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:28:53.726562 6944 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 23:28:53.729593 6977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:53.731138 6944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 23:28:53.731355 6944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/wal
uuid: "030a6b9f073b4e8b827b8d8d166cf958"
format_stamp: "Formatted at 2026-08-10 23:28:47 on dist-test-slave-rmhg"
server_key: "59f1ec654178333a512a97c89c12d1d7"
server_key_iv: "e5686c028fa802da1c7dabdd9325c175"
server_key_version: "encryptionkey@0"
I20260810 23:28:53.732511 6944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:28:53.748116 6944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:28:53.752653 6944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:28:53.752916 6944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:28:53.754024 6944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:28:53.755424 6944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:28:53.755525 6944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:53.755614 6944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:28:53.755668 6944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:53.789028 6944 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.66:44945
I20260810 23:28:53.789103 7151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.66:44945 every 8 connection(s)
I20260810 23:28:53.790578 6944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
I20260810 23:28:53.791477 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 6944
I20260810 23:28:53.792198 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.67:37491
--local_ip_for_outbound_sockets=127.14.175.67
--tserver_master_addrs=127.14.175.126:42557
--webserver_port=33487
--webserver_interface=127.14.175.67
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:42173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:28:53.796087 7155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:53.793912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.66:33801)
0810 23:28:53.794816 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:53.795112 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":354,"threads_started":1}
W20260810 23:28:53.796768 7152 heartbeater.cc:644] Failed to heartbeat to 127.14.175.126:42557 (0 consecutive failures): Network error: Failed to ping master at 127.14.175.126:42557: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
I20260810 23:28:53.797714 7155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:53.797360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.66:39299)
0810 23:28:53.797514 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:53.797588 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 23:28:53.798575 7155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:53.798232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.66:34189)
0810 23:28:53.798377 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:53.798464 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 23:28:53.798852 7152 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 23:28:54.147742 7157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:28:54.148079 7157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:28:54.148207 7157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:28:54.155882 7157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:28:54.156020 7157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:28:54.156184 7157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.67
I20260810 23:28:54.166603 7157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:42173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.67:37491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.175.67
--webserver_port=33487
--tserver_master_addrs=127.14.175.126:42557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.67
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:28:54.168063 7157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:28:54.169596 7157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:28:54.185575 7165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:54.185770 7167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:28:54.186323 7164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:28:54.186833 7157 server_base.cc:1034] running on GCE node
I20260810 23:28:54.187672 7157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:28:54.188769 7157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:28:54.189985 7157 hybrid_clock.cc:648] HybridClock initialized: now 1786404534189922 us; error 52 us; skew 500 ppm
I20260810 23:28:54.190412 7157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:28:54 dist-test-slave-rmhg krb5kdc[4807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404534, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:28:54.194578 7157 init.cc:377] Logged in from keytab as kudu/127.14.175.67@KRBTEST.COM (short username kudu)
I20260810 23:28:54.196698 7157 webserver.cc:460] Webserver started at http://127.14.175.67:33487/ using document root <none> and password file <none>
I20260810 23:28:54.197773 7157 fs_manager.cc:359] Metadata directory not provided
I20260810 23:28:54.197934 7157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:28:54.205964 7157 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.003s
I20260810 23:28:54.210969 7177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:54.212970 7157 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260810 23:28:54.213243 7157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/wal
uuid: "0ae6b985366f4d47914c40302b102253"
format_stamp: "Formatted at 2026-08-10 23:28:48 on dist-test-slave-rmhg"
server_key: "a60d61e517106eb7f148903bc164cc02"
server_key_iv: "20126f69f44dfc76e4e820faf3c1c8d4"
server_key_version: "encryptionkey@0"
I20260810 23:28:54.214951 7157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:28:54.255402 7157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:28:54.259584 7157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:28:54.259840 7157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:28:54.260852 7157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:28:54.262275 7157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:28:54.262378 7157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:54.262466 7157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:28:54.262521 7157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:28:54.295324 7157 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.67:37491
I20260810 23:28:54.295418 7352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.67:37491 every 8 connection(s)
I20260810 23:28:54.296876 7157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
I20260810 23:28:54.302194 7356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:54.300040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.67:50283)
0810 23:28:54.300864 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:54.301101 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":319,"threads_started":1}
I20260810 23:28:54.302467 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 7157
W20260810 23:28:54.303171 7354 heartbeater.cc:644] Failed to heartbeat to 127.14.175.126:42557 (0 consecutive failures): Network error: Failed to ping master at 127.14.175.126:42557: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
I20260810 23:28:54.304594 7356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:54.303821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.67:45681)
0810 23:28:54.304380 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:54.304451 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 23:28:54.305589 7356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:54.305142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.67:54573)
0810 23:28:54.305330 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:54.305432 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 23:28:54.305891 7354 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 23:28:54.320840 7359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:54.320013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.65:42383)
0810 23:28:54.320496 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:54.320593 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":207,"threads_started":1}
I20260810 23:28:54.800671 7360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:54.799605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.66:53559)
0810 23:28:54.800204 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:54.800385 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":268,"threads_started":1}
I20260810 23:28:55.146194 7362 raft_consensus.cc:482] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:28:55.146629 7362 raft_consensus.cc:504] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030a6b9f073b4e8b827b8d8d166cf958" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44945 } } peers { permanent_uuid: "0ae6b985366f4d47914c40302b102253" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37491 } } peers { permanent_uuid: "c2025f4fdf524504bfe241da57e32cbf" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35659 } }
I20260810 23:28:55.148363 7362 leader_election.cc:288] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 030a6b9f073b4e8b827b8d8d166cf958 (127.14.175.66:44945), 0ae6b985366f4d47914c40302b102253 (127.14.175.67:37491)
Aug 10 23:28:55 dist-test-slave-rmhg krb5kdc[4807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404533, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for kudu/127.14.175.66@KRBTEST.COM
Aug 10 23:28:55 dist-test-slave-rmhg krb5kdc[4807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404533, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for kudu/127.14.175.67@KRBTEST.COM
I20260810 23:28:55.183743 7371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:55.149043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:52667 (local address 127.14.175.67:37491)
0810 23:28:55.162173 (+ 13130us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:55.162189 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:55.162240 (+ 51us) server_negotiation.cc:407] Connection header received
0810 23:28:55.164339 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:55.164351 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:55.164847 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:55.165257 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:55.168088 (+ 2831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:55.169367 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:55.170910 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:55.171603 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:55.171812 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:55.176386 (+ 4574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:55.176394 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:55.176409 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:55.176457 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:55.180242 (+ 3785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:55.180923 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:55.180930 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:55.180936 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:55.181109 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:55.181539 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:55.181543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:55.181545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:55.182044 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:55.182218 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:55.182950 (+ 732us) server_negotiation.cc:299] Negotiation successful
0810 23:28:55.183307 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12929,"thread_start_us":12788,"threads_started":1}
I20260810 23:28:55.183771 7372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:55.149085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:49587 (local address 127.14.175.66:44945)
0810 23:28:55.161362 (+ 12277us) server_negotiation.cc:206] Beginning negotiation
0810 23:28:55.161389 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 23:28:55.162857 (+ 1468us) server_negotiation.cc:407] Connection header received
0810 23:28:55.164549 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:28:55.164559 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:28:55.164931 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:28:55.165293 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:28:55.167706 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:55.169051 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:55.170917 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:28:55.171615 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:28:55.171815 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:55.176236 (+ 4421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:28:55.176245 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:28:55.176263 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:28:55.176314 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:28:55.179730 (+ 3416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:55.180615 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:55.180624 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:55.180630 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:55.180815 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:28:55.181293 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:28:55.181297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:28:55.181299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:28:55.181900 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:28:55.182094 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0810 23:28:55.182950 (+ 856us) server_negotiation.cc:299] Negotiation successful
0810 23:28:55.183368 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11962,"thread_start_us":11724,"threads_started":1}
I20260810 23:28:55.183784 7370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:55.148867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:44945 (local address 127.14.175.65:49587)
0810 23:28:55.162679 (+ 13812us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:55.162698 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:55.162848 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:55.165489 (+ 2641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:55.165502 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:55.166552 (+ 1050us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:28:55.167402 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:55.167430 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:55.169295 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:55.169299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:55.170746 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:55.170761 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:55.171891 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:55.171901 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:55.172251 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:55.173071 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:28:55.173096 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:28:55.175870 (+ 2774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:28:55.179922 (+ 4052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:55.179932 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:55.179946 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:55.180446 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:55.180936 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:55.180940 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:55.180942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:55.181134 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:55.182046 (+ 912us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:28:55.182055 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:28:55.182377 (+ 322us) client_negotiation.cc:769] Sending connection context
0810 23:28:55.182646 (+ 269us) client_negotiation.cc:240] Negotiation successful
0810 23:28:55.183285 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13678,"mutex_wait_us":122,"spinlock_wait_cycles":28544,"thread_start_us":13448,"threads_started":1}
I20260810 23:28:55.184075 7367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:55.148759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:37491 (local address 127.14.175.65:52667)
0810 23:28:55.161726 (+ 12967us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:55.161809 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0810 23:28:55.162549 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:28:55.165489 (+ 2940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:28:55.165501 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:28:55.166524 (+ 1023us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:28:55.167401 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:55.167429 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:55.169543 (+ 2114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:55.169547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:55.170745 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:28:55.170759 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:28:55.171839 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:28:55.171845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:28:55.172251 (+ 406us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:28:55.173083 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:28:55.173109 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:28:55.176068 (+ 2959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:28:55.180402 (+ 4334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:55.180408 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:55.180413 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:55.180770 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:55.181250 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:28:55.181255 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:28:55.181259 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:28:55.181420 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:28:55.182201 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:28:55.182206 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:28:55.182378 (+ 172us) client_negotiation.cc:769] Sending connection context
0810 23:28:55.182646 (+ 268us) client_negotiation.cc:240] Negotiation successful
0810 23:28:55.183467 (+ 821us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12806,"mutex_wait_us":421,"spinlock_wait_cycles":32128,"thread_start_us":12617,"threads_started":1}
W20260810 23:28:55.187263 6758 leader_election.cc:341] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 030a6b9f073b4e8b827b8d8d166cf958 (127.14.175.66:44945): Not found: Tablet not found: 31d7643768824cdda4325ff4092ce767
W20260810 23:28:55.187531 6760 leader_election.cc:341] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0ae6b985366f4d47914c40302b102253 (127.14.175.67:37491): Not found: Tablet not found: 31d7643768824cdda4325ff4092ce767
I20260810 23:28:55.187681 6760 leader_election.cc:302] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2025f4fdf524504bfe241da57e32cbf; no voters: 030a6b9f073b4e8b827b8d8d166cf958, 0ae6b985366f4d47914c40302b102253
I20260810 23:28:55.188077 7362 raft_consensus.cc:2726] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:28:55.307425 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:55.306448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.67:36275)
0810 23:28:55.307020 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:55.307181 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":310,"threads_started":1}
I20260810 23:28:55.322458 7367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:55.321990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.65:52377)
0810 23:28:55.322185 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:55.322308 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 23:28:55.802549 7377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:55.801722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.66:44181)
0810 23:28:55.802238 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:55.802354 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":229,"threads_started":1}
I20260810 23:28:56.309219 7382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:56.308443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.67:41871)
0810 23:28:56.308919 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:56.309011 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":222,"threads_started":1}
I20260810 23:28:56.324066 7384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:56.323331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.65:53439)
0810 23:28:56.323791 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:56.323875 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":219,"threads_started":1}
I20260810 23:28:56.804365 7389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:56.803582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.66:33033)
0810 23:28:56.804071 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:56.804179 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":252,"threads_started":1}
I20260810 23:28:57.130424 7391 raft_consensus.cc:482] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:28:57.130642 7391 raft_consensus.cc:504] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030a6b9f073b4e8b827b8d8d166cf958" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44945 } } peers { permanent_uuid: "0ae6b985366f4d47914c40302b102253" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37491 } } peers { permanent_uuid: "c2025f4fdf524504bfe241da57e32cbf" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35659 } }
I20260810 23:28:57.131348 7391 leader_election.cc:288] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 030a6b9f073b4e8b827b8d8d166cf958 (127.14.175.66:44945), 0ae6b985366f4d47914c40302b102253 (127.14.175.67:37491)
W20260810 23:28:57.132491 6760 leader_election.cc:341] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0ae6b985366f4d47914c40302b102253 (127.14.175.67:37491): Not found: Tablet not found: 31d7643768824cdda4325ff4092ce767
W20260810 23:28:57.132692 6758 leader_election.cc:341] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 030a6b9f073b4e8b827b8d8d166cf958 (127.14.175.66:44945): Not found: Tablet not found: 31d7643768824cdda4325ff4092ce767
I20260810 23:28:57.132856 6758 leader_election.cc:302] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2025f4fdf524504bfe241da57e32cbf; no voters: 030a6b9f073b4e8b827b8d8d166cf958, 0ae6b985366f4d47914c40302b102253
I20260810 23:28:57.133188 7391 raft_consensus.cc:2726] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:28:57.311102 7392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:57.310240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.67:53867)
0810 23:28:57.310787 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:57.310898 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":271,"threads_started":1}
I20260810 23:28:57.325973 7393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:57.325049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.65:38141)
0810 23:28:57.325644 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:57.325780 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":321,"threads_started":1}
I20260810 23:28:57.806397 7394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:57.805549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.66:44453)
0810 23:28:57.806057 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:57.806156 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":249,"threads_started":1}
I20260810 23:28:58.312965 7395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:58.312128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.67:34477)
0810 23:28:58.312672 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:58.312761 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":280,"threads_started":1}
I20260810 23:28:58.327821 7397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:58.326975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.65:39937)
0810 23:28:58.327506 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:58.327608 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":253,"threads_started":1}
I20260810 23:28:58.808287 7398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:58.807422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.66:35373)
0810 23:28:58.807970 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:58.808105 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":313,"threads_started":1}
I20260810 23:28:59.314761 7399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:59.313957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.67:42301)
0810 23:28:59.314483 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:59.314580 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":276,"threads_started":1}
I20260810 23:28:59.329527 7401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:59.328736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.65:59501)
0810 23:28:59.329269 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:59.329361 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":269,"threads_started":1}
I20260810 23:28:59.810240 7402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:28:59.809316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.66:41561)
0810 23:28:59.809904 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0810 23:28:59.810007 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":277,"threads_started":1}
I20260810 23:29:00.316449 7405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:00.315714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.67:37579)
0810 23:29:00.316181 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:00.316269 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":245,"threads_started":1}
I20260810 23:29:00.331470 7407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:00.330523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.65:33301)
0810 23:29:00.331063 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:00.331206 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":249,"threads_started":1}
I20260810 23:29:00.336139 7409 raft_consensus.cc:482] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:29:00.336344 7409 raft_consensus.cc:504] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030a6b9f073b4e8b827b8d8d166cf958" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44945 } } peers { permanent_uuid: "0ae6b985366f4d47914c40302b102253" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37491 } } peers { permanent_uuid: "c2025f4fdf524504bfe241da57e32cbf" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35659 } }
I20260810 23:29:00.337069 7409 leader_election.cc:288] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 030a6b9f073b4e8b827b8d8d166cf958 (127.14.175.66:44945), 0ae6b985366f4d47914c40302b102253 (127.14.175.67:37491)
W20260810 23:29:00.338145 6758 leader_election.cc:341] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 030a6b9f073b4e8b827b8d8d166cf958 (127.14.175.66:44945): Not found: Tablet not found: 31d7643768824cdda4325ff4092ce767
W20260810 23:29:00.338325 6760 leader_election.cc:341] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0ae6b985366f4d47914c40302b102253 (127.14.175.67:37491): Not found: Tablet not found: 31d7643768824cdda4325ff4092ce767
I20260810 23:29:00.338452 6760 leader_election.cc:302] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2025f4fdf524504bfe241da57e32cbf; no voters: 030a6b9f073b4e8b827b8d8d166cf958, 0ae6b985366f4d47914c40302b102253
I20260810 23:29:00.338682 7409 raft_consensus.cc:2726] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:29:00.812282 7410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:00.811357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.66:42865)
0810 23:29:00.811971 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:00.812084 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":282,"threads_started":1}
I20260810 23:29:01.318241 7412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:01.317432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.67:54521)
0810 23:29:01.317943 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:01.318030 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":223,"threads_started":1}
I20260810 23:29:01.333266 7413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:01.332460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.65:50037)
0810 23:29:01.332943 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:01.333041 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":248,"threads_started":1}
I20260810 23:29:01.813974 7415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:01.813263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.66:52827)
0810 23:29:01.813688 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:01.813780 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":209,"threads_started":1}
I20260810 23:29:02.320236 7416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:02.319286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.67:42395)
0810 23:29:02.319890 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:02.319994 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":311,"threads_started":1}
I20260810 23:29:02.334915 7417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:02.334128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.65:50467)
0810 23:29:02.334642 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:02.334733 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":259,"threads_started":1}
I20260810 23:29:02.815896 7420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:02.815046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.66:34425)
0810 23:29:02.815626 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:02.815725 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":285,"threads_started":1}
I20260810 23:29:03.322046 7421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:03.321262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.67:45137)
0810 23:29:03.321794 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:03.321886 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":233,"threads_started":1}
I20260810 23:29:03.336714 7422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:03.335921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.65:51197)
0810 23:29:03.336399 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:03.336484 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":255,"threads_started":1}
I20260810 23:29:03.817909 7423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:03.817071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.66:51977)
0810 23:29:03.817609 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:03.817710 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":290,"threads_started":1}
I20260810 23:29:04.323596 7426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:04.322900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.67:48627)
0810 23:29:04.323331 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:04.323415 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":228,"threads_started":1}
I20260810 23:29:04.338464 7428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:04.337717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.65:55859)
0810 23:29:04.338177 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:04.338268 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":225,"threads_started":1}
I20260810 23:29:04.564100 7429 raft_consensus.cc:482] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:29:04.564337 7429 raft_consensus.cc:504] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030a6b9f073b4e8b827b8d8d166cf958" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44945 } } peers { permanent_uuid: "0ae6b985366f4d47914c40302b102253" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37491 } } peers { permanent_uuid: "c2025f4fdf524504bfe241da57e32cbf" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35659 } }
I20260810 23:29:04.565029 7429 leader_election.cc:288] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 030a6b9f073b4e8b827b8d8d166cf958 (127.14.175.66:44945), 0ae6b985366f4d47914c40302b102253 (127.14.175.67:37491)
W20260810 23:29:04.566179 6758 leader_election.cc:341] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 030a6b9f073b4e8b827b8d8d166cf958 (127.14.175.66:44945): Not found: Tablet not found: 31d7643768824cdda4325ff4092ce767
W20260810 23:29:04.566354 6760 leader_election.cc:341] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0ae6b985366f4d47914c40302b102253 (127.14.175.67:37491): Not found: Tablet not found: 31d7643768824cdda4325ff4092ce767
I20260810 23:29:04.566473 6760 leader_election.cc:302] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2025f4fdf524504bfe241da57e32cbf; no voters: 030a6b9f073b4e8b827b8d8d166cf958, 0ae6b985366f4d47914c40302b102253
I20260810 23:29:04.566699 7429 raft_consensus.cc:2726] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:29:04.820144 7430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:04.819191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.66:60561)
0810 23:29:04.819832 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:04.819939 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":317,"threads_started":1}
I20260810 23:29:05.325451 7431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:05.324579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.67:33119)
0810 23:29:05.325121 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:05.325279 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":277,"threads_started":1}
I20260810 23:29:05.340343 7432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:05.339448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.65:52235)
0810 23:29:05.340015 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:05.340133 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":332,"threads_started":1}
I20260810 23:29:05.821933 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:05.821132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.66:47359)
0810 23:29:05.821634 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:05.821749 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":221,"threads_started":1}
I20260810 23:29:06.327374 7436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:06.326594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.67:51657)
0810 23:29:06.327089 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:06.327187 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":241,"threads_started":1}
I20260810 23:29:06.342420 7439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:06.341559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.65:53931)
0810 23:29:06.342113 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:06.342226 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":276,"threads_started":1}
I20260810 23:29:06.823719 7440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:06.822910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.66:59197)
0810 23:29:06.823414 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:06.823516 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":258,"threads_started":1}
I20260810 23:29:07.329200 7441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:07.328419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.67:56461)
0810 23:29:07.328874 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:07.328973 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":230,"threads_started":1}
I20260810 23:29:07.344208 7445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:07.343426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.65:59293)
0810 23:29:07.343907 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:07.344017 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":264,"threads_started":1}
I20260810 23:29:07.825706 7446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:07.824773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.66:47029)
0810 23:29:07.825348 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:07.825461 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":263,"threads_started":1}
I20260810 23:29:08.331036 7449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:08.330190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.67:47465)
0810 23:29:08.330697 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:08.330811 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":276,"threads_started":1}
I20260810 23:29:08.346670 7450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:08.345730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.65:39647)
0810 23:29:08.346349 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:08.346455 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":322,"threads_started":1}
I20260810 23:29:08.437232 7452 raft_consensus.cc:482] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:29:08.437490 7452 raft_consensus.cc:504] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "030a6b9f073b4e8b827b8d8d166cf958" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44945 } } peers { permanent_uuid: "0ae6b985366f4d47914c40302b102253" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37491 } } peers { permanent_uuid: "c2025f4fdf524504bfe241da57e32cbf" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 35659 } }
I20260810 23:29:08.438225 7452 leader_election.cc:288] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 030a6b9f073b4e8b827b8d8d166cf958 (127.14.175.66:44945), 0ae6b985366f4d47914c40302b102253 (127.14.175.67:37491)
W20260810 23:29:08.439332 6760 leader_election.cc:341] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0ae6b985366f4d47914c40302b102253 (127.14.175.67:37491): Not found: Tablet not found: 31d7643768824cdda4325ff4092ce767
W20260810 23:29:08.439558 6758 leader_election.cc:341] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 030a6b9f073b4e8b827b8d8d166cf958 (127.14.175.66:44945): Not found: Tablet not found: 31d7643768824cdda4325ff4092ce767
I20260810 23:29:08.439706 6758 leader_election.cc:302] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2025f4fdf524504bfe241da57e32cbf; no voters: 030a6b9f073b4e8b827b8d8d166cf958, 0ae6b985366f4d47914c40302b102253
I20260810 23:29:08.439965 7452 raft_consensus.cc:2726] T 31d7643768824cdda4325ff4092ce767 P c2025f4fdf524504bfe241da57e32cbf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:29:08.827755 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:08.826944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.66:48601)
0810 23:29:08.827428 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:08.827562 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":244,"threads_started":1}
I20260810 23:29:09.315124 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 6663
I20260810 23:29:09.333036 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:09.332220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.67:50471)
0810 23:29:09.332767 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:09.332861 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":310,"threads_started":1}
I20260810 23:29:09.336571 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:09.316416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44136 (local address 127.14.175.65:35659)
0810 23:29:09.316916 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0810 23:29:09.316923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:29:09.316953 (+ 30us) server_negotiation.cc:407] Connection header received
0810 23:29:09.317015 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:29:09.317022 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:29:09.317292 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:29:09.317452 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:29:09.318825 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:09.319827 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:09.321272 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:09.321962 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:09.322063 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:09.330355 (+ 8292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:29:09.330374 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:29:09.330393 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:29:09.330443 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:29:09.333733 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:09.334585 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:09.334593 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:09.334595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:09.334749 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:09.335049 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:09.335052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:09.335054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:09.335458 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:29:09.335703 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0810 23:29:09.336012 (+ 309us) server_negotiation.cc:299] Negotiation successful
0810 23:29:09.336297 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":249,"threads_started":1}
I20260810 23:29:09.337958 6934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:29:09.427726 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:09.426751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:42557 (local address 127.14.175.65:53035)
0810 23:29:09.427265 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:09.427401 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:42557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":242,"threads_started":1}
I20260810 23:29:09.427760 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 6663
I20260810 23:29:09.452344 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 6944
I20260810 23:29:09.466284 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:09.453139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58784 (local address 127.14.175.66:44945)
0810 23:29:09.453711 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0810 23:29:09.453716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:29:09.453733 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:29:09.453792 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:29:09.453795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:29:09.453855 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:29:09.453952 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:29:09.455375 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:09.456219 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:09.457572 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:09.458024 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:09.458141 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:09.460840 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:29:09.460845 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:29:09.460849 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:29:09.460886 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:29:09.463504 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:09.464225 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:09.464230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:09.464232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:09.464370 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:09.464763 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:09.464767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:09.464769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:09.465078 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:29:09.465378 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0810 23:29:09.465652 (+ 274us) server_negotiation.cc:299] Negotiation successful
0810 23:29:09.465883 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":280,"threads_started":1}
I20260810 23:29:09.467048 7142 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:29:09.552285 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 6944
I20260810 23:29:09.579358 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 7157
I20260810 23:29:09.594179 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:09.580393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45742 (local address 127.14.175.67:37491)
0810 23:29:09.580982 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0810 23:29:09.580987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:29:09.581003 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:29:09.581060 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:29:09.581063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:29:09.581122 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:29:09.581254 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:29:09.582769 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:09.583600 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:09.584891 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:09.585457 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:09.585565 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:09.588565 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:29:09.588572 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:29:09.588575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:29:09.588617 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:29:09.591356 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:09.592108 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:09.592111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:09.592114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:09.592263 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:09.592706 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:09.592709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:09.592711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:09.592993 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:29:09.593266 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0810 23:29:09.593659 (+ 393us) server_negotiation.cc:299] Negotiation successful
0810 23:29:09.593869 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":311,"threads_started":1}
I20260810 23:29:09.594965 7347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:29:09.679176 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 7157
2026-08-10T23:29:09Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24921 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:29:09 dist-test-slave-rmhg krb5kdc[7469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:29:09 dist-test-slave-rmhg krb5kdc[7469](info): set up 2 sockets
Aug 10 23:29:09 dist-test-slave-rmhg krb5kdc[7469](info): commencing operation
krb5kdc: starting...
W20260810 23:29:10.933526 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.207s user 0.002s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:29:11 dist-test-slave-rmhg krb5kdc[7469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:29:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:29:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
I20260810 23:29:11.235168 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:45843
--webserver_interface=127.14.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:42347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:45843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:29:11.590785 7489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:29:11.591118 7489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:29:11.591252 7489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:29:11.598631 7489 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 23:29:11.598752 7489 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 23:29:11.598811 7489 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 23:29:11.598881 7489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:29:11.598951 7489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:29:11.599051 7489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:29:11.599114 7489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:29:11.599190 7489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:29:11.609449 7489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:42347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:45843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:45843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:29:11.610849 7489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:29:11.612262 7489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:29:11.624737 7500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:29:11.624744 7505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:29:11.624737 7501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:29:11.625627 7489 server_base.cc:1034] running on GCE node
I20260810 23:29:11.626447 7489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:29:11.628041 7489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:29:11.629272 7489 hybrid_clock.cc:648] HybridClock initialized: now 1786404551629187 us; error 62 us; skew 500 ppm
I20260810 23:29:11.629621 7489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:29:11 dist-test-slave-rmhg krb5kdc[7469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404551, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:29:11.632864 7489 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:29:11.634528 7489 webserver.cc:460] Webserver started at http://127.14.175.126:34573/ using document root <none> and password file <none>
I20260810 23:29:11.635267 7489 fs_manager.cc:359] Metadata directory not provided
I20260810 23:29:11.635378 7489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:29:11.635706 7489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:29:11.638850 7489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "3f3e6979d111499fb5b7805eabf0241e"
format_stamp: "Formatted at 2026-08-10 23:29:11 on dist-test-slave-rmhg"
server_key: "a1129d012a7ae976ff06d2ec6dde2c51"
server_key_iv: "6d6bf709d15e134d04e2b1560dcab157"
server_key_version: "encryptionkey@0"
I20260810 23:29:11.639612 7489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "3f3e6979d111499fb5b7805eabf0241e"
format_stamp: "Formatted at 2026-08-10 23:29:11 on dist-test-slave-rmhg"
server_key: "a1129d012a7ae976ff06d2ec6dde2c51"
server_key_iv: "6d6bf709d15e134d04e2b1560dcab157"
server_key_version: "encryptionkey@0"
I20260810 23:29:11.645299 7489 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260810 23:29:11.648981 7518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:29:11.650432 7489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:29:11.650610 7489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "3f3e6979d111499fb5b7805eabf0241e"
format_stamp: "Formatted at 2026-08-10 23:29:11 on dist-test-slave-rmhg"
server_key: "a1129d012a7ae976ff06d2ec6dde2c51"
server_key_iv: "6d6bf709d15e134d04e2b1560dcab157"
server_key_version: "encryptionkey@0"
I20260810 23:29:11.650870 7489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:29:11.669523 7489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:29:11.673857 7489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:29:11.674147 7489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:29:11.692960 7489 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:45843
I20260810 23:29:11.692982 7609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:45843 every 8 connection(s)
I20260810 23:29:11.694530 7489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:29:11.694954 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 7489
I20260810 23:29:11.695211 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:29:11.696069 15037 external_mini_cluster.cc:1428] Setting key 8b38b72b0050c35cd52cf8c647f4067b
I20260810 23:29:11.699398 7614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 23:29:11 dist-test-slave-rmhg krb5kdc[7469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:29:11.713886 7614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e: Bootstrap starting.
I20260810 23:29:11.717442 7614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e: Neither blocks nor log segments found. Creating new log.
I20260810 23:29:11.718851 7614 log.cc:824] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:29:11.721853 7620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:11.698227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55132 (local address 127.14.175.126:45843)
0810 23:29:11.699200 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0810 23:29:11.699217 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:29:11.699600 (+ 383us) server_negotiation.cc:407] Connection header received
0810 23:29:11.701469 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:29:11.701502 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:29:11.701920 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:29:11.702299 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:29:11.704337 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:11.705881 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:11.707972 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:11.708718 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:11.708868 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:11.714024 (+ 5156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:29:11.714037 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:29:11.714058 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:29:11.714103 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:29:11.717184 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:11.718213 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:11.718221 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:11.718228 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:11.718404 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:11.718875 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:11.718879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:11.718881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:11.719551 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:29:11.719915 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0810 23:29:11.720313 (+ 398us) server_negotiation.cc:299] Negotiation successful
0810 23:29:11.720765 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":393,"threads_started":1}
I20260810 23:29:11.722236 7614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e: No bootstrap required, opened a new log
I20260810 23:29:11.730002 7614 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3e6979d111499fb5b7805eabf0241e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 45843 } }
I20260810 23:29:11.730443 7614 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:29:11.730572 7614 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f3e6979d111499fb5b7805eabf0241e, State: Initialized, Role: FOLLOWER
I20260810 23:29:11.731309 7614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3e6979d111499fb5b7805eabf0241e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 45843 } }
I20260810 23:29:11.731549 7614 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:29:11.731664 7614 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:29:11.731801 7614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:29:11.734107 7614 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3e6979d111499fb5b7805eabf0241e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 45843 } }
I20260810 23:29:11.734611 7614 leader_election.cc:302] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f3e6979d111499fb5b7805eabf0241e; no voters:
I20260810 23:29:11.735337 7614 leader_election.cc:288] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:29:11.735473 7622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:29:11.736204 7622 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [term 1 LEADER]: Becoming Leader. State: Replica: 3f3e6979d111499fb5b7805eabf0241e, State: Running, Role: LEADER
I20260810 23:29:11.736855 7622 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3e6979d111499fb5b7805eabf0241e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 45843 } }
I20260810 23:29:11.737809 7614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:29:11.739991 7623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f3e6979d111499fb5b7805eabf0241e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3e6979d111499fb5b7805eabf0241e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 45843 } } }
I20260810 23:29:11.741536 7623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [sys.catalog]: This master's current role is: LEADER
I20260810 23:29:11.740082 7624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f3e6979d111499fb5b7805eabf0241e. Latest consensus state: current_term: 1 leader_uuid: "3f3e6979d111499fb5b7805eabf0241e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3e6979d111499fb5b7805eabf0241e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 45843 } } }
I20260810 23:29:11.745609 7624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [sys.catalog]: This master's current role is: LEADER
I20260810 23:29:11.746621 7634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:29:11.752260 7634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:29:11.762535 7634 catalog_manager.cc:1347] Generated new cluster ID: 506ab82d48304dcd94becd57e18afe2a
I20260810 23:29:11.762705 7634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:29:11.772004 7634 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:29:11.773478 7634 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:29:11.785594 7634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e: Generated new TSK 0
I20260810 23:29:11.786520 7634 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
I20260810 23:29:11.865063 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.65:0
--local_ip_for_outbound_sockets=127.14.175.65
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:45843
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:42347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:29:12.225203 7651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:29:12.225524 7651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:29:12.225660 7651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:29:12.233223 7651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:29:12.233368 7651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:29:12.233529 7651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.65
I20260810 23:29:12.244143 7651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:42347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:45843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.65
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:29:12.245713 7651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:29:12.247148 7651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:29:12.261471 7661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:29:12.261439 7662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:29:12.261427 7664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:29:12.262193 7651 server_base.cc:1034] running on GCE node
I20260810 23:29:12.262908 7651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:29:12.263924 7651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:29:12.265199 7651 hybrid_clock.cc:648] HybridClock initialized: now 1786404552265101 us; error 46 us; skew 500 ppm
I20260810 23:29:12.265553 7651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:29:12 dist-test-slave-rmhg krb5kdc[7469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404552, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:29:12.268805 7651 init.cc:377] Logged in from keytab as kudu/127.14.175.65@KRBTEST.COM (short username kudu)
I20260810 23:29:12.270349 7651 webserver.cc:460] Webserver started at http://127.14.175.65:46523/ using document root <none> and password file <none>
I20260810 23:29:12.271072 7651 fs_manager.cc:359] Metadata directory not provided
I20260810 23:29:12.271198 7651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:29:12.271504 7651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:29:12.274637 7651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/data/instance:
uuid: "4875e7ba636a4c9fa099a90d2abb0e7c"
format_stamp: "Formatted at 2026-08-10 23:29:12 on dist-test-slave-rmhg"
server_key: "cc6fb69dac1033ba3e7baba662a4ba6e"
server_key_iv: "03536a959811b9e3f943b60185fbc337"
server_key_version: "encryptionkey@0"
I20260810 23:29:12.275490 7651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance:
uuid: "4875e7ba636a4c9fa099a90d2abb0e7c"
format_stamp: "Formatted at 2026-08-10 23:29:12 on dist-test-slave-rmhg"
server_key: "cc6fb69dac1033ba3e7baba662a4ba6e"
server_key_iv: "03536a959811b9e3f943b60185fbc337"
server_key_version: "encryptionkey@0"
I20260810 23:29:12.280921 7651 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 23:29:12.284873 7673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:29:12.286445 7651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:29:12.286609 7651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/wal
uuid: "4875e7ba636a4c9fa099a90d2abb0e7c"
format_stamp: "Formatted at 2026-08-10 23:29:12 on dist-test-slave-rmhg"
server_key: "cc6fb69dac1033ba3e7baba662a4ba6e"
server_key_iv: "03536a959811b9e3f943b60185fbc337"
server_key_version: "encryptionkey@0"
I20260810 23:29:12.286835 7651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:29:12.307071 7651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:29:12.311796 7651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:29:12.312107 7651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:29:12.313241 7651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:29:12.314844 7651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:29:12.314931 7651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:29:12.315066 7651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:29:12.315124 7651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:29:12.347352 7651 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.65:43593
I20260810 23:29:12.347398 7867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.65:43593 every 8 connection(s)
I20260810 23:29:12.348699 7651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
I20260810 23:29:12.356763 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 7651
I20260810 23:29:12.356967 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance
I20260810 23:29:12.357784 15037 external_mini_cluster.cc:1428] Setting key e6459cb7863a19901451818c488e9044
Aug 10 23:29:12 dist-test-slave-rmhg krb5kdc[7469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404552, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:29:12.370826 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:12.351317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:42841 (local address 127.14.175.126:45843)
0810 23:29:12.351793 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0810 23:29:12.351799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:29:12.352794 (+ 995us) server_negotiation.cc:407] Connection header received
0810 23:29:12.354637 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:29:12.354641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:29:12.354741 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:29:12.354837 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:29:12.356500 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:12.357322 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:12.359450 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:12.360010 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:12.360115 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:12.364311 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:29:12.364316 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:29:12.364319 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:29:12.364353 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:29:12.367545 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:12.368378 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:12.368384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:12.368387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:12.368540 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:12.368956 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:12.368960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:12.368963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:12.369325 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:29:12.369462 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 23:29:12.370251 (+ 789us) server_negotiation.cc:299] Negotiation successful
0810 23:29:12.370474 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":223,"threads_started":1}
I20260810 23:29:12.371814 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:12.351899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.65:42841)
0810 23:29:12.352625 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:12.352674 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 23:29:12.354389 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:29:12.355104 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:29:12.355113 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:29:12.355655 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:29:12.356321 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:12.356347 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:12.357498 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:12.357504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:12.359305 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:12.359319 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:12.360265 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:12.360274 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:12.360581 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:12.361302 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:29:12.361336 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:29:12.364057 (+ 2721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:29:12.367701 (+ 3644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:29:12.367711 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:29:12.367728 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:29:12.368250 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:29:12.368633 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:29:12.368637 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:29:12.368640 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:29:12.368857 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:29:12.369431 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:29:12.369438 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:29:12.369856 (+ 418us) client_negotiation.cc:769] Sending connection context
0810 23:29:12.370126 (+ 270us) client_negotiation.cc:240] Negotiation successful
0810 23:29:12.370606 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":266,"threads_started":1}
I20260810 23:29:12.373623 7868 heartbeater.cc:347] Connected to a master server at 127.14.175.126:45843
I20260810 23:29:12.373968 7868 heartbeater.cc:464] Registering TS with master...
I20260810 23:29:12.374897 7868 heartbeater.cc:511] Master 127.14.175.126:45843 requested a full tablet report, sending...
I20260810 23:29:12.377703 7550 ts_manager.cc:194] Registered new tserver with Master: 4875e7ba636a4c9fa099a90d2abb0e7c (127.14.175.65:43593)
I20260810 23:29:12.379822 7550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.65@KRBTEST.COM'} at 127.14.175.65:42841
WARNING: no policy specified for kudu/127.14.175.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
I20260810 23:29:12.426458 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.66:0
--local_ip_for_outbound_sockets=127.14.175.66
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:45843
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:42347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:29:12.771896 7876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:29:12.772236 7876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:29:12.772373 7876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:29:12.780140 7876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:29:12.780300 7876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:29:12.780459 7876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.66
I20260810 23:29:12.790882 7876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:42347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:45843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.66
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:29:12.792302 7876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:29:12.793779 7876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:29:12.807317 7882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:29:12.807286 7890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:29:12.807348 7886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:29:12.808331 7876 server_base.cc:1034] running on GCE node
I20260810 23:29:12.808967 7876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:29:12.810019 7876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:29:12.811232 7876 hybrid_clock.cc:648] HybridClock initialized: now 1786404552811185 us; error 38 us; skew 500 ppm
I20260810 23:29:12.811554 7876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:29:12 dist-test-slave-rmhg krb5kdc[7469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404552, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:29:12.814843 7876 init.cc:377] Logged in from keytab as kudu/127.14.175.66@KRBTEST.COM (short username kudu)
I20260810 23:29:12.816401 7876 webserver.cc:460] Webserver started at http://127.14.175.66:36815/ using document root <none> and password file <none>
I20260810 23:29:12.817095 7876 fs_manager.cc:359] Metadata directory not provided
I20260810 23:29:12.817253 7876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:29:12.817567 7876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:29:12.820654 7876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/data/instance:
uuid: "dc053cfca0964ebf87c75bd4dee18bc7"
format_stamp: "Formatted at 2026-08-10 23:29:12 on dist-test-slave-rmhg"
server_key: "d2aa37f0ba78c8eda5f8708686eadfa9"
server_key_iv: "099366519a0caba015482782ac9b470a"
server_key_version: "encryptionkey@0"
I20260810 23:29:12.821508 7876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance:
uuid: "dc053cfca0964ebf87c75bd4dee18bc7"
format_stamp: "Formatted at 2026-08-10 23:29:12 on dist-test-slave-rmhg"
server_key: "d2aa37f0ba78c8eda5f8708686eadfa9"
server_key_iv: "099366519a0caba015482782ac9b470a"
server_key_version: "encryptionkey@0"
I20260810 23:29:12.827014 7876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 23:29:12.830682 7901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:29:12.832118 7876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260810 23:29:12.832279 7876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/wal
uuid: "dc053cfca0964ebf87c75bd4dee18bc7"
format_stamp: "Formatted at 2026-08-10 23:29:12 on dist-test-slave-rmhg"
server_key: "d2aa37f0ba78c8eda5f8708686eadfa9"
server_key_iv: "099366519a0caba015482782ac9b470a"
server_key_version: "encryptionkey@0"
I20260810 23:29:12.832470 7876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:29:12.858304 7876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:29:12.862990 7876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:29:12.863303 7876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:29:12.864391 7876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:29:12.865926 7876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:29:12.866010 7876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:29:12.866132 7876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:29:12.866184 7876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:29:12.898203 7876 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.66:37175
I20260810 23:29:12.898276 8080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.66:37175 every 8 connection(s)
I20260810 23:29:12.899508 7876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
I20260810 23:29:12.905807 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 7876
I20260810 23:29:12.906023 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance
I20260810 23:29:12.906590 15037 external_mini_cluster.cc:1428] Setting key f8801dda9052e2c78fd25aacacc0f583
Aug 10 23:29:12 dist-test-slave-rmhg krb5kdc[7469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404552, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:29:12.921312 8083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:12.902101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:39601 (local address 127.14.175.126:45843)
0810 23:29:12.905863 (+ 3762us) server_negotiation.cc:206] Beginning negotiation
0810 23:29:12.905868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:29:12.905891 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:29:12.907127 (+ 1236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:29:12.907133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:29:12.907214 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:29:12.907331 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:29:12.909404 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:12.910135 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:12.911741 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:12.912186 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:12.912297 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:12.915768 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:29:12.915772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:29:12.915775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:29:12.915808 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:29:12.918238 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:12.918956 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:12.918960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:12.918962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:12.919101 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:12.919513 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:12.919517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:12.919519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:12.919780 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:29:12.919912 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 23:29:12.920672 (+ 760us) server_negotiation.cc:299] Negotiation successful
0810 23:29:12.920902 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3640,"thread_start_us":3533,"threads_started":1}
I20260810 23:29:12.922322 8085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:12.902789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.66:39601)
0810 23:29:12.903993 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:12.904090 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0810 23:29:12.906724 (+ 2634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:29:12.907686 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:29:12.907699 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:29:12.908542 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:29:12.909231 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:12.909255 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:12.910297 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:12.910301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:12.911598 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:12.911611 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:12.912415 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:12.912423 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:12.912661 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:12.913362 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:29:12.913385 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:29:12.915548 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:29:12.918373 (+ 2825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:29:12.918383 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:29:12.918402 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:29:12.918834 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:29:12.919207 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:29:12.919212 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:29:12.919216 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:29:12.919398 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:29:12.919911 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:29:12.919920 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:29:12.920258 (+ 338us) client_negotiation.cc:769] Sending connection context
0810 23:29:12.920521 (+ 263us) client_negotiation.cc:240] Negotiation successful
0810 23:29:12.921087 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"thread_start_us":451,"threads_started":1}
I20260810 23:29:12.923839 8081 heartbeater.cc:347] Connected to a master server at 127.14.175.126:45843
I20260810 23:29:12.924208 8081 heartbeater.cc:464] Registering TS with master...
I20260810 23:29:12.925215 8081 heartbeater.cc:511] Master 127.14.175.126:45843 requested a full tablet report, sending...
I20260810 23:29:12.927013 7550 ts_manager.cc:194] Registered new tserver with Master: dc053cfca0964ebf87c75bd4dee18bc7 (127.14.175.66:37175)
I20260810 23:29:12.928128 7550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.66@KRBTEST.COM'} at 127.14.175.66:39601
WARNING: no policy specified for kudu/127.14.175.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
I20260810 23:29:12.971803 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.67:0
--local_ip_for_outbound_sockets=127.14.175.67
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:45843
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:42347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:29:13.318565 8089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:29:13.318876 8089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:29:13.318996 8089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:29:13.326423 8089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:29:13.326558 8089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:29:13.326712 8089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.67
I20260810 23:29:13.336874 8089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:42347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:45843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.67
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:29:13.338392 8089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:29:13.339836 8089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:29:13.354048 8097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:29:13.354051 8100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:29:13.354033 8098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:29:13.354681 8089 server_base.cc:1034] running on GCE node
I20260810 23:29:13.355337 8089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:29:13.356325 8089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:29:13.357584 8089 hybrid_clock.cc:648] HybridClock initialized: now 1786404553357516 us; error 60 us; skew 500 ppm
I20260810 23:29:13.357951 8089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:29:13 dist-test-slave-rmhg krb5kdc[7469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404553, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:29:13.361296 8089 init.cc:377] Logged in from keytab as kudu/127.14.175.67@KRBTEST.COM (short username kudu)
I20260810 23:29:13.362778 8089 webserver.cc:460] Webserver started at http://127.14.175.67:37915/ using document root <none> and password file <none>
I20260810 23:29:13.363471 8089 fs_manager.cc:359] Metadata directory not provided
I20260810 23:29:13.363574 8089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:29:13.363863 8089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:29:13.366909 8089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/data/instance:
uuid: "2f41eb4f81424f35b0f19a66421a8cd0"
format_stamp: "Formatted at 2026-08-10 23:29:13 on dist-test-slave-rmhg"
server_key: "dde0da0c43fcf4d8c1a7178d0c4bd29f"
server_key_iv: "9d9e62a9d4195e91536755027ee856ff"
server_key_version: "encryptionkey@0"
I20260810 23:29:13.367640 8089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance:
uuid: "2f41eb4f81424f35b0f19a66421a8cd0"
format_stamp: "Formatted at 2026-08-10 23:29:13 on dist-test-slave-rmhg"
server_key: "dde0da0c43fcf4d8c1a7178d0c4bd29f"
server_key_iv: "9d9e62a9d4195e91536755027ee856ff"
server_key_version: "encryptionkey@0"
I20260810 23:29:13.372946 8089 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:29:13.376446 8114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:29:13.377974 8089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:29:13.378149 8089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/wal
uuid: "2f41eb4f81424f35b0f19a66421a8cd0"
format_stamp: "Formatted at 2026-08-10 23:29:13 on dist-test-slave-rmhg"
server_key: "dde0da0c43fcf4d8c1a7178d0c4bd29f"
server_key_iv: "9d9e62a9d4195e91536755027ee856ff"
server_key_version: "encryptionkey@0"
I20260810 23:29:13.378346 8089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:29:13.383648 7868 heartbeater.cc:503] Master 127.14.175.126:45843 was elected leader, sending a full tablet report...
I20260810 23:29:13.407028 8089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:29:13.411355 8089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:29:13.411659 8089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:29:13.412670 8089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:29:13.414105 8089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:29:13.414192 8089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:29:13.414314 8089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:29:13.414368 8089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:29:13.445230 8089 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.67:36617
I20260810 23:29:13.445274 8298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.67:36617 every 8 connection(s)
I20260810 23:29:13.446511 8089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
I20260810 23:29:13.451306 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 8089
I20260810 23:29:13.451503 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance
I20260810 23:29:13.452006 15037 external_mini_cluster.cc:1428] Setting key f7caf02669d6def2eb8d3da72661f8b5
Aug 10 23:29:13 dist-test-slave-rmhg krb5kdc[7469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404553, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:29:13.467960 8303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:13.449034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:38847 (local address 127.14.175.126:45843)
0810 23:29:13.449518 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0810 23:29:13.449523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:29:13.450427 (+ 904us) server_negotiation.cc:407] Connection header received
0810 23:29:13.452503 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:29:13.452510 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:29:13.452603 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:29:13.452705 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:29:13.454561 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:13.455541 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:13.457221 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:13.457690 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:13.457790 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:13.461652 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:29:13.461658 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:29:13.461661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:29:13.461702 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:29:13.464381 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:13.465266 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:13.465270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:13.465273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:13.465400 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:13.465925 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:13.465930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:13.465932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:13.466318 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:29:13.466473 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 23:29:13.467411 (+ 938us) server_negotiation.cc:299] Negotiation successful
0810 23:29:13.467652 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":249,"threads_started":1}
I20260810 23:29:13.469060 8304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:13.449559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.67:38847)
0810 23:29:13.450270 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:13.450311 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 23:29:13.452182 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:29:13.453011 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:29:13.453024 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:29:13.453690 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:29:13.454377 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:13.454407 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:13.455722 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:13.455726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:13.457031 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:13.457049 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:13.457899 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:13.457904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:13.458136 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:13.458810 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:29:13.458836 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:29:13.461353 (+ 2517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:29:13.464589 (+ 3236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:29:13.464599 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:29:13.464615 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:29:13.465103 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:29:13.465533 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:29:13.465537 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:29:13.465540 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:29:13.465765 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:29:13.466488 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:29:13.466497 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:29:13.466986 (+ 489us) client_negotiation.cc:769] Sending connection context
0810 23:29:13.467274 (+ 288us) client_negotiation.cc:240] Negotiation successful
0810 23:29:13.467850 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":272,"threads_started":1}
I20260810 23:29:13.470485 8301 heartbeater.cc:347] Connected to a master server at 127.14.175.126:45843
I20260810 23:29:13.470796 8301 heartbeater.cc:464] Registering TS with master...
I20260810 23:29:13.471498 8301 heartbeater.cc:511] Master 127.14.175.126:45843 requested a full tablet report, sending...
I20260810 23:29:13.472942 7550 ts_manager.cc:194] Registered new tserver with Master: 2f41eb4f81424f35b0f19a66421a8cd0 (127.14.175.67:36617)
I20260810 23:29:13.473798 7550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.67@KRBTEST.COM'} at 127.14.175.67:38847
I20260810 23:29:13.480531 15037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:29:13.498183 8303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:13.485138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55140 (local address 127.14.175.126:45843)
0810 23:29:13.485329 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 23:29:13.485333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:29:13.485796 (+ 463us) server_negotiation.cc:407] Connection header received
0810 23:29:13.485948 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:29:13.485952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:29:13.486017 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:29:13.486099 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:29:13.487570 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:13.488320 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:13.489802 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:13.490244 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:13.490355 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:13.493000 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:29:13.493006 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:29:13.493009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:29:13.493050 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:29:13.495732 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:13.496424 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:13.496427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:13.496430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:13.496552 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:13.496849 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:13.496851 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:13.496853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:13.497220 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:29:13.497410 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0810 23:29:13.497748 (+ 338us) server_negotiation.cc:299] Negotiation successful
0810 23:29:13.497941 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 23:29:13.502506 7550 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:29:13.509963 7550 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:29:13.548789 8319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:13.522891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:37175 (local address 127.0.0.1:56438)
0810 23:29:13.539702 (+ 16811us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:13.539728 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 23:29:13.539880 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:29:13.541093 (+ 1213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:29:13.541098 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:29:13.541123 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:29:13.541670 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:13.541690 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:13.544166 (+ 2476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:13.544171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:13.546206 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:13.546219 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:13.547666 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:13.547673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:13.547887 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:13.547948 (+ 61us) client_negotiation.cc:769] Sending connection context
0810 23:29:13.548087 (+ 139us) client_negotiation.cc:240] Negotiation successful
0810 23:29:13.548314 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16431,"mutex_wait_us":4,"thread_start_us":12058,"threads_started":1}
I20260810 23:29:13.549654 8324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:13.523376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56438 (local address 127.14.175.66:37175)
0810 23:29:13.540461 (+ 17085us) server_negotiation.cc:206] Beginning negotiation
0810 23:29:13.540469 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:29:13.540499 (+ 30us) server_negotiation.cc:407] Connection header received
0810 23:29:13.540593 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:29:13.540600 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:29:13.540853 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:29:13.541509 (+ 656us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:29:13.542534 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:13.544014 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:13.546363 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:13.547390 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:13.547530 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:13.547752 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0810 23:29:13.549103 (+ 1351us) server_negotiation.cc:299] Negotiation successful
0810 23:29:13.549335 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16930,"thread_start_us":16779,"threads_started":1}
I20260810 23:29:13.549670 8323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:13.522889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41872 (local address 127.14.175.65:43593)
0810 23:29:13.539528 (+ 16639us) server_negotiation.cc:206] Beginning negotiation
0810 23:29:13.539540 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:29:13.539578 (+ 38us) server_negotiation.cc:407] Connection header received
0810 23:29:13.539709 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:29:13.539720 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:29:13.540028 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:29:13.540544 (+ 516us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:29:13.541392 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:13.542982 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:13.545516 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:13.547073 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:13.547239 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:13.547489 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0810 23:29:13.549231 (+ 1742us) server_negotiation.cc:299] Negotiation successful
0810 23:29:13.549421 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16398,"thread_start_us":17338,"threads_started":1}
I20260810 23:29:13.550153 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:13.522663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:43593 (local address 127.0.0.1:41872)
0810 23:29:13.535439 (+ 12776us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:13.535496 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 23:29:13.535747 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:29:13.540589 (+ 4842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:29:13.540598 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:29:13.540633 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:29:13.541111 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:13.541132 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:13.543156 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:13.543162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:13.545307 (+ 2145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:13.545336 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:13.547322 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:13.547330 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:13.548387 (+ 1057us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:13.548422 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 23:29:13.548509 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 23:29:13.548644 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12442,"mutex_wait_us":2,"thread_start_us":16747,"threads_started":1}
I20260810 23:29:13.551182 8330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:13.524141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51708 (local address 127.14.175.67:36617)
0810 23:29:13.541353 (+ 17212us) server_negotiation.cc:206] Beginning negotiation
0810 23:29:13.541363 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:29:13.542378 (+ 1015us) server_negotiation.cc:407] Connection header received
0810 23:29:13.542599 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:29:13.542607 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:29:13.542828 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:29:13.543014 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:29:13.543877 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:13.545470 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:13.547861 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:13.549716 (+ 1855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:13.549857 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:13.550092 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 23:29:13.550659 (+ 567us) server_negotiation.cc:299] Negotiation successful
0810 23:29:13.550836 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16976,"thread_start_us":16773,"threads_started":1}
I20260810 23:29:13.551605 8326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:13.523451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:36617 (local address 127.0.0.1:51708)
0810 23:29:13.542232 (+ 18781us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:13.542280 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:29:13.542400 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:29:13.543323 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:29:13.543326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:29:13.543349 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:29:13.543701 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:13.543708 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:13.545611 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:13.545616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:13.547612 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:13.547625 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:13.550247 (+ 2622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:13.550255 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:13.550421 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:13.550451 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 23:29:13.550545 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 23:29:13.550689 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18660,"thread_start_us":18522,"threads_started":1}
I20260810 23:29:13.553972 7982 tablet_service.cc:1467] Processing CreateTablet for tablet 44f7ca59496a46f8ba4507e482235eed (DEFAULT_TABLE table=test-table [id=cda23cf97ca64034ad2ff14a43793d9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:29:13.553972 7766 tablet_service.cc:1467] Processing CreateTablet for tablet 44f7ca59496a46f8ba4507e482235eed (DEFAULT_TABLE table=test-table [id=cda23cf97ca64034ad2ff14a43793d9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:29:13.554634 8206 tablet_service.cc:1467] Processing CreateTablet for tablet 44f7ca59496a46f8ba4507e482235eed (DEFAULT_TABLE table=test-table [id=cda23cf97ca64034ad2ff14a43793d9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:29:13.555495 7766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44f7ca59496a46f8ba4507e482235eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:29:13.555513 7982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44f7ca59496a46f8ba4507e482235eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:29:13.556725 8206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44f7ca59496a46f8ba4507e482235eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:29:13.568686 8331 tablet_bootstrap.cc:492] T 44f7ca59496a46f8ba4507e482235eed P 2f41eb4f81424f35b0f19a66421a8cd0: Bootstrap starting.
I20260810 23:29:13.572144 8331 tablet_bootstrap.cc:654] T 44f7ca59496a46f8ba4507e482235eed P 2f41eb4f81424f35b0f19a66421a8cd0: Neither blocks nor log segments found. Creating new log.
I20260810 23:29:13.573349 8331 log.cc:824] T 44f7ca59496a46f8ba4507e482235eed P 2f41eb4f81424f35b0f19a66421a8cd0: Log is configured to *not* fsync() on all Append() calls
I20260810 23:29:13.574376 8333 tablet_bootstrap.cc:492] T 44f7ca59496a46f8ba4507e482235eed P dc053cfca0964ebf87c75bd4dee18bc7: Bootstrap starting.
I20260810 23:29:13.575065 8332 tablet_bootstrap.cc:492] T 44f7ca59496a46f8ba4507e482235eed P 4875e7ba636a4c9fa099a90d2abb0e7c: Bootstrap starting.
I20260810 23:29:13.578504 8331 tablet_bootstrap.cc:492] T 44f7ca59496a46f8ba4507e482235eed P 2f41eb4f81424f35b0f19a66421a8cd0: No bootstrap required, opened a new log
I20260810 23:29:13.578812 8333 tablet_bootstrap.cc:654] T 44f7ca59496a46f8ba4507e482235eed P dc053cfca0964ebf87c75bd4dee18bc7: Neither blocks nor log segments found. Creating new log.
I20260810 23:29:13.578845 8331 ts_tablet_manager.cc:1397] T 44f7ca59496a46f8ba4507e482235eed P 2f41eb4f81424f35b0f19a66421a8cd0: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260810 23:29:13.579002 8332 tablet_bootstrap.cc:654] T 44f7ca59496a46f8ba4507e482235eed P 4875e7ba636a4c9fa099a90d2abb0e7c: Neither blocks nor log segments found. Creating new log.
I20260810 23:29:13.579999 8333 log.cc:824] T 44f7ca59496a46f8ba4507e482235eed P dc053cfca0964ebf87c75bd4dee18bc7: Log is configured to *not* fsync() on all Append() calls
I20260810 23:29:13.580333 8332 log.cc:824] T 44f7ca59496a46f8ba4507e482235eed P 4875e7ba636a4c9fa099a90d2abb0e7c: Log is configured to *not* fsync() on all Append() calls
I20260810 23:29:13.588004 8332 tablet_bootstrap.cc:492] T 44f7ca59496a46f8ba4507e482235eed P 4875e7ba636a4c9fa099a90d2abb0e7c: No bootstrap required, opened a new log
I20260810 23:29:13.588208 8333 tablet_bootstrap.cc:492] T 44f7ca59496a46f8ba4507e482235eed P dc053cfca0964ebf87c75bd4dee18bc7: No bootstrap required, opened a new log
I20260810 23:29:13.588316 8332 ts_tablet_manager.cc:1397] T 44f7ca59496a46f8ba4507e482235eed P 4875e7ba636a4c9fa099a90d2abb0e7c: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260810 23:29:13.588490 8333 ts_tablet_manager.cc:1397] T 44f7ca59496a46f8ba4507e482235eed P dc053cfca0964ebf87c75bd4dee18bc7: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260810 23:29:13.589941 8331 raft_consensus.cc:348] T 44f7ca59496a46f8ba4507e482235eed P 2f41eb4f81424f35b0f19a66421a8cd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4875e7ba636a4c9fa099a90d2abb0e7c" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 43593 } } peers { permanent_uuid: "dc053cfca0964ebf87c75bd4dee18bc7" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 37175 } } peers { permanent_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 36617 } }
I20260810 23:29:13.590426 8331 raft_consensus.cc:374] T 44f7ca59496a46f8ba4507e482235eed P 2f41eb4f81424f35b0f19a66421a8cd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:29:13.590534 8331 raft_consensus.cc:729] T 44f7ca59496a46f8ba4507e482235eed P 2f41eb4f81424f35b0f19a66421a8cd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f41eb4f81424f35b0f19a66421a8cd0, State: Initialized, Role: FOLLOWER
I20260810 23:29:13.591281 8331 consensus_queue.cc:260] T 44f7ca59496a46f8ba4507e482235eed P 2f41eb4f81424f35b0f19a66421a8cd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4875e7ba636a4c9fa099a90d2abb0e7c" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 43593 } } peers { permanent_uuid: "dc053cfca0964ebf87c75bd4dee18bc7" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 37175 } } peers { permanent_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 36617 } }
I20260810 23:29:13.592609 8301 heartbeater.cc:503] Master 127.14.175.126:45843 was elected leader, sending a full tablet report...
I20260810 23:29:13.593349 8331 ts_tablet_manager.cc:1428] T 44f7ca59496a46f8ba4507e482235eed P 2f41eb4f81424f35b0f19a66421a8cd0: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260810 23:29:13.598075 8333 raft_consensus.cc:348] T 44f7ca59496a46f8ba4507e482235eed P dc053cfca0964ebf87c75bd4dee18bc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4875e7ba636a4c9fa099a90d2abb0e7c" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 43593 } } peers { permanent_uuid: "dc053cfca0964ebf87c75bd4dee18bc7" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 37175 } } peers { permanent_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 36617 } }
I20260810 23:29:13.598590 8333 raft_consensus.cc:374] T 44f7ca59496a46f8ba4507e482235eed P dc053cfca0964ebf87c75bd4dee18bc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:29:13.598719 8333 raft_consensus.cc:729] T 44f7ca59496a46f8ba4507e482235eed P dc053cfca0964ebf87c75bd4dee18bc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc053cfca0964ebf87c75bd4dee18bc7, State: Initialized, Role: FOLLOWER
I20260810 23:29:13.599380 8333 consensus_queue.cc:260] T 44f7ca59496a46f8ba4507e482235eed P dc053cfca0964ebf87c75bd4dee18bc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4875e7ba636a4c9fa099a90d2abb0e7c" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 43593 } } peers { permanent_uuid: "dc053cfca0964ebf87c75bd4dee18bc7" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 37175 } } peers { permanent_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 36617 } }
I20260810 23:29:13.600528 8081 heartbeater.cc:503] Master 127.14.175.126:45843 was elected leader, sending a full tablet report...
I20260810 23:29:13.600442 8332 raft_consensus.cc:348] T 44f7ca59496a46f8ba4507e482235eed P 4875e7ba636a4c9fa099a90d2abb0e7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4875e7ba636a4c9fa099a90d2abb0e7c" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 43593 } } peers { permanent_uuid: "dc053cfca0964ebf87c75bd4dee18bc7" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 37175 } } peers { permanent_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 36617 } }
I20260810 23:29:13.600931 8332 raft_consensus.cc:374] T 44f7ca59496a46f8ba4507e482235eed P 4875e7ba636a4c9fa099a90d2abb0e7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:29:13.601039 8332 raft_consensus.cc:729] T 44f7ca59496a46f8ba4507e482235eed P 4875e7ba636a4c9fa099a90d2abb0e7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4875e7ba636a4c9fa099a90d2abb0e7c, State: Initialized, Role: FOLLOWER
I20260810 23:29:13.601125 8333 ts_tablet_manager.cc:1428] T 44f7ca59496a46f8ba4507e482235eed P dc053cfca0964ebf87c75bd4dee18bc7: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260810 23:29:13.601702 8332 consensus_queue.cc:260] T 44f7ca59496a46f8ba4507e482235eed P 4875e7ba636a4c9fa099a90d2abb0e7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4875e7ba636a4c9fa099a90d2abb0e7c" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 43593 } } peers { permanent_uuid: "dc053cfca0964ebf87c75bd4dee18bc7" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 37175 } } peers { permanent_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 36617 } }
I20260810 23:29:13.603413 8332 ts_tablet_manager.cc:1428] T 44f7ca59496a46f8ba4507e482235eed P 4875e7ba636a4c9fa099a90d2abb0e7c: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260810 23:29:13.651513 8082 tablet.cc:2369] T 44f7ca59496a46f8ba4507e482235eed P dc053cfca0964ebf87c75bd4dee18bc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:29:13.698398 8302 tablet.cc:2369] T 44f7ca59496a46f8ba4507e482235eed P 2f41eb4f81424f35b0f19a66421a8cd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:29:13.851667 7869 tablet.cc:2369] T 44f7ca59496a46f8ba4507e482235eed P 4875e7ba636a4c9fa099a90d2abb0e7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:29:13.855293 8339 raft_consensus.cc:482] T 44f7ca59496a46f8ba4507e482235eed P 2f41eb4f81424f35b0f19a66421a8cd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:29:13.855619 8339 raft_consensus.cc:504] T 44f7ca59496a46f8ba4507e482235eed P 2f41eb4f81424f35b0f19a66421a8cd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4875e7ba636a4c9fa099a90d2abb0e7c" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 43593 } } peers { permanent_uuid: "dc053cfca0964ebf87c75bd4dee18bc7" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 37175 } } peers { permanent_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 36617 } }
I20260810 23:29:13.857467 8339 leader_election.cc:288] T 44f7ca59496a46f8ba4507e482235eed P 2f41eb4f81424f35b0f19a66421a8cd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4875e7ba636a4c9fa099a90d2abb0e7c (127.14.175.65:43593), dc053cfca0964ebf87c75bd4dee18bc7 (127.14.175.66:37175)
I20260810 23:29:13.864526 8323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:13.857886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:58999 (local address 127.14.175.65:43593)
0810 23:29:13.858126 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 23:29:13.858133 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:29:13.858524 (+ 391us) server_negotiation.cc:407] Connection header received
0810 23:29:13.858609 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:29:13.858614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:29:13.858694 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:29:13.858776 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:29:13.859190 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:13.860181 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:13.862541 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:13.863719 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:13.863886 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:13.863962 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 23:29:13.864194 (+ 232us) server_negotiation.cc:299] Negotiation successful
0810 23:29:13.864318 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 23:29:13.864612 8304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:13.857904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:43593 (local address 127.14.175.67:58999)
0810 23:29:13.858268 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:13.858288 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:29:13.858433 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:29:13.858834 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:29:13.858839 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:29:13.858870 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:29:13.859056 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:13.859064 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:13.860316 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:13.860320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:13.862351 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:13.862373 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:13.863885 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:13.863889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:13.864035 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:13.864062 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 23:29:13.864151 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 23:29:13.864319 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217}
I20260810 23:29:13.865000 8324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:13.858090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:46603 (local address 127.14.175.66:37175)
0810 23:29:13.858271 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 23:29:13.858276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:29:13.858882 (+ 606us) server_negotiation.cc:407] Connection header received
0810 23:29:13.859003 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:29:13.859007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:29:13.859065 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:29:13.859143 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:29:13.859653 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:13.860544 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:13.862826 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:13.864040 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:13.864166 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:13.864232 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 23:29:13.864595 (+ 363us) server_negotiation.cc:299] Negotiation successful
0810 23:29:13.864763 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 23:29:13.865314 8343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:13.857942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:37175 (local address 127.14.175.67:46603)
0810 23:29:13.858767 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:13.858784 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:29:13.858892 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:29:13.859175 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:29:13.859179 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:29:13.859199 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:29:13.859466 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:13.859473 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:13.860698 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:13.860703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:13.862657 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:13.862668 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:13.864241 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:13.864246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:13.864411 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:13.864446 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 23:29:13.864548 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 23:29:13.864987 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":487,"threads_started":1}
I20260810 23:29:13.865600 7796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f7ca59496a46f8ba4507e482235eed" candidate_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4875e7ba636a4c9fa099a90d2abb0e7c" is_pre_election: true
I20260810 23:29:13.866201 7796 raft_consensus.cc:2454] T 44f7ca59496a46f8ba4507e482235eed P 4875e7ba636a4c9fa099a90d2abb0e7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f41eb4f81424f35b0f19a66421a8cd0 in term 0.
I20260810 23:29:13.866341 8015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f7ca59496a46f8ba4507e482235eed" candidate_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc053cfca0964ebf87c75bd4dee18bc7" is_pre_election: true
I20260810 23:29:13.866838 8015 raft_consensus.cc:2454] T 44f7ca59496a46f8ba4507e482235eed P dc053cfca0964ebf87c75bd4dee18bc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f41eb4f81424f35b0f19a66421a8cd0 in term 0.
I20260810 23:29:13.866940 8119 leader_election.cc:302] T 44f7ca59496a46f8ba4507e482235eed P 2f41eb4f81424f35b0f19a66421a8cd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f41eb4f81424f35b0f19a66421a8cd0, 4875e7ba636a4c9fa099a90d2abb0e7c; no voters:
I20260810 23:29:13.867372 8339 raft_consensus.cc:2781] T 44f7ca59496a46f8ba4507e482235eed P 2f41eb4f81424f35b0f19a66421a8cd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:29:13.867532 8339 raft_consensus.cc:482] T 44f7ca59496a46f8ba4507e482235eed P 2f41eb4f81424f35b0f19a66421a8cd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:29:13.867647 8339 raft_consensus.cc:3037] T 44f7ca59496a46f8ba4507e482235eed P 2f41eb4f81424f35b0f19a66421a8cd0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:29:13.869841 8339 raft_consensus.cc:504] T 44f7ca59496a46f8ba4507e482235eed P 2f41eb4f81424f35b0f19a66421a8cd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4875e7ba636a4c9fa099a90d2abb0e7c" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 43593 } } peers { permanent_uuid: "dc053cfca0964ebf87c75bd4dee18bc7" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 37175 } } peers { permanent_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 36617 } }
I20260810 23:29:13.870409 8339 leader_election.cc:288] T 44f7ca59496a46f8ba4507e482235eed P 2f41eb4f81424f35b0f19a66421a8cd0 [CANDIDATE]: Term 1 election: Requested vote from peers 4875e7ba636a4c9fa099a90d2abb0e7c (127.14.175.65:43593), dc053cfca0964ebf87c75bd4dee18bc7 (127.14.175.66:37175)
I20260810 23:29:13.870877 8015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f7ca59496a46f8ba4507e482235eed" candidate_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc053cfca0964ebf87c75bd4dee18bc7"
I20260810 23:29:13.870903 7796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f7ca59496a46f8ba4507e482235eed" candidate_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4875e7ba636a4c9fa099a90d2abb0e7c"
I20260810 23:29:13.871061 8015 raft_consensus.cc:3037] T 44f7ca59496a46f8ba4507e482235eed P dc053cfca0964ebf87c75bd4dee18bc7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:29:13.871078 7796 raft_consensus.cc:3037] T 44f7ca59496a46f8ba4507e482235eed P 4875e7ba636a4c9fa099a90d2abb0e7c [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:29:13.873282 7796 raft_consensus.cc:2454] T 44f7ca59496a46f8ba4507e482235eed P 4875e7ba636a4c9fa099a90d2abb0e7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f41eb4f81424f35b0f19a66421a8cd0 in term 1.
I20260810 23:29:13.873306 8015 raft_consensus.cc:2454] T 44f7ca59496a46f8ba4507e482235eed P dc053cfca0964ebf87c75bd4dee18bc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f41eb4f81424f35b0f19a66421a8cd0 in term 1.
I20260810 23:29:13.873699 8116 leader_election.cc:302] T 44f7ca59496a46f8ba4507e482235eed P 2f41eb4f81424f35b0f19a66421a8cd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f41eb4f81424f35b0f19a66421a8cd0, dc053cfca0964ebf87c75bd4dee18bc7; no voters:
I20260810 23:29:13.873994 8339 raft_consensus.cc:2781] T 44f7ca59496a46f8ba4507e482235eed P 2f41eb4f81424f35b0f19a66421a8cd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:29:13.874445 8339 raft_consensus.cc:686] T 44f7ca59496a46f8ba4507e482235eed P 2f41eb4f81424f35b0f19a66421a8cd0 [term 1 LEADER]: Becoming Leader. State: Replica: 2f41eb4f81424f35b0f19a66421a8cd0, State: Running, Role: LEADER
I20260810 23:29:13.874881 8339 consensus_queue.cc:237] T 44f7ca59496a46f8ba4507e482235eed P 2f41eb4f81424f35b0f19a66421a8cd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4875e7ba636a4c9fa099a90d2abb0e7c" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 43593 } } peers { permanent_uuid: "dc053cfca0964ebf87c75bd4dee18bc7" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 37175 } } peers { permanent_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 36617 } }
I20260810 23:29:13.879328 7550 catalog_manager.cc:5495] T 44f7ca59496a46f8ba4507e482235eed P 2f41eb4f81424f35b0f19a66421a8cd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f41eb4f81424f35b0f19a66421a8cd0 (127.14.175.67). New cstate: current_term: 1 leader_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4875e7ba636a4c9fa099a90d2abb0e7c" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 43593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc053cfca0964ebf87c75bd4dee18bc7" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 37175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 36617 } health_report { overall_health: HEALTHY } } }
I20260810 23:29:13.905769 8330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:13.899644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51710 (local address 127.14.175.67:36617)
0810 23:29:13.899846 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 23:29:13.899852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:29:13.899875 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:29:13.900095 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:29:13.900101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:29:13.900212 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:29:13.900372 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:29:13.900677 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:13.901774 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:13.903315 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:13.903931 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:13.904049 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:13.904516 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:29:13.905197 (+ 681us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:29:13.905338 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 23:29:13.905382 (+ 44us) server_negotiation.cc:299] Negotiation successful
0810 23:29:13.905464 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 23:29:13.914857 8015 raft_consensus.cc:1261] T 44f7ca59496a46f8ba4507e482235eed P dc053cfca0964ebf87c75bd4dee18bc7 [term 1 FOLLOWER]: Refusing update from remote peer 2f41eb4f81424f35b0f19a66421a8cd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:29:13.914863 7796 raft_consensus.cc:1261] T 44f7ca59496a46f8ba4507e482235eed P 4875e7ba636a4c9fa099a90d2abb0e7c [term 1 FOLLOWER]: Refusing update from remote peer 2f41eb4f81424f35b0f19a66421a8cd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:29:13.915884 8347 consensus_queue.cc:1035] T 44f7ca59496a46f8ba4507e482235eed P 2f41eb4f81424f35b0f19a66421a8cd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc053cfca0964ebf87c75bd4dee18bc7" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:29:13.916214 8339 consensus_queue.cc:1035] T 44f7ca59496a46f8ba4507e482235eed P 2f41eb4f81424f35b0f19a66421a8cd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4875e7ba636a4c9fa099a90d2abb0e7c" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 43593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:29:13.928658 8350 mvcc.cc:205] Tried to move back new op lower bound from 7317113052825001984 to 7317113052675227648. Current Snapshot: MvccSnapshot[applied={T|T < 7317113052825001984}]
I20260810 23:29:13.935573 8354 mvcc.cc:205] Tried to move back new op lower bound from 7317113052825001984 to 7317113052675227648. Current Snapshot: MvccSnapshot[applied={T|T < 7317113052825001984}]
I20260810 23:29:13.945041 7548 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:29:13.945439 7548 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:29:13.949642 7548 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e: Sending DeleteTablet for 3 replicas of tablet 44f7ca59496a46f8ba4507e482235eed
I20260810 23:29:13.950872 7766 tablet_service.cc:1514] Processing DeleteTablet for tablet 44f7ca59496a46f8ba4507e482235eed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:29:13 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:41872
I20260810 23:29:13.951071 7982 tablet_service.cc:1514] Processing DeleteTablet for tablet 44f7ca59496a46f8ba4507e482235eed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:29:13 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:56438
I20260810 23:29:13.952121 8368 tablet_replica.cc:330] T 44f7ca59496a46f8ba4507e482235eed P dc053cfca0964ebf87c75bd4dee18bc7: stopping tablet replica
I20260810 23:29:13.952114 8206 tablet_service.cc:1514] Processing DeleteTablet for tablet 44f7ca59496a46f8ba4507e482235eed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:29:13 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:51708
I20260810 23:29:13.952899 8368 raft_consensus.cc:2229] T 44f7ca59496a46f8ba4507e482235eed P dc053cfca0964ebf87c75bd4dee18bc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:29:13.953090 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 7651
I20260810 23:29:13.953677 8368 raft_consensus.cc:2258] T 44f7ca59496a46f8ba4507e482235eed P dc053cfca0964ebf87c75bd4dee18bc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:29:13.956341 8368 ts_tablet_manager.cc:1905] T 44f7ca59496a46f8ba4507e482235eed P dc053cfca0964ebf87c75bd4dee18bc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:29:13.964762 8368 ts_tablet_manager.cc:1918] T 44f7ca59496a46f8ba4507e482235eed P dc053cfca0964ebf87c75bd4dee18bc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:29:13.965137 8368 log.cc:1196] T 44f7ca59496a46f8ba4507e482235eed P dc053cfca0964ebf87c75bd4dee18bc7: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/wal/wals/44f7ca59496a46f8ba4507e482235eed
I20260810 23:29:13.965879 8368 ts_tablet_manager.cc:1939] T 44f7ca59496a46f8ba4507e482235eed P dc053cfca0964ebf87c75bd4dee18bc7: Deleting consensus metadata
I20260810 23:29:13.968245 7520 catalog_manager.cc:4841] TS dc053cfca0964ebf87c75bd4dee18bc7 (127.14.175.66:37175): tablet 44f7ca59496a46f8ba4507e482235eed (table test-table [id=cda23cf97ca64034ad2ff14a43793d9d]) successfully deleted
I20260810 23:29:13.970748 8369 tablet_replica.cc:330] T 44f7ca59496a46f8ba4507e482235eed P 2f41eb4f81424f35b0f19a66421a8cd0: stopping tablet replica
I20260810 23:29:13.971395 8369 raft_consensus.cc:2229] T 44f7ca59496a46f8ba4507e482235eed P 2f41eb4f81424f35b0f19a66421a8cd0 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:29:13.972234 8369 raft_consensus.cc:2258] T 44f7ca59496a46f8ba4507e482235eed P 2f41eb4f81424f35b0f19a66421a8cd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:29:13.973201 8367 tablet_replica.cc:330] T 44f7ca59496a46f8ba4507e482235eed P 4875e7ba636a4c9fa099a90d2abb0e7c: stopping tablet replica
I20260810 23:29:13.973733 8367 raft_consensus.cc:2229] T 44f7ca59496a46f8ba4507e482235eed P 4875e7ba636a4c9fa099a90d2abb0e7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:29:13.974371 8367 raft_consensus.cc:2258] T 44f7ca59496a46f8ba4507e482235eed P 4875e7ba636a4c9fa099a90d2abb0e7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:29:13.975819 8369 ts_tablet_manager.cc:1905] T 44f7ca59496a46f8ba4507e482235eed P 2f41eb4f81424f35b0f19a66421a8cd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:29:13.976397 8367 ts_tablet_manager.cc:1905] T 44f7ca59496a46f8ba4507e482235eed P 4875e7ba636a4c9fa099a90d2abb0e7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:29:13.983639 8367 ts_tablet_manager.cc:1918] T 44f7ca59496a46f8ba4507e482235eed P 4875e7ba636a4c9fa099a90d2abb0e7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:29:13.983611 8369 ts_tablet_manager.cc:1918] T 44f7ca59496a46f8ba4507e482235eed P 2f41eb4f81424f35b0f19a66421a8cd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Aug 10 23:29:13 dist-test-slave-rmhg krb5kdc[7469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.65@KRBTEST.COM
I20260810 23:29:13.983824 8367 log.cc:1196] T 44f7ca59496a46f8ba4507e482235eed P 4875e7ba636a4c9fa099a90d2abb0e7c: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/wal/wals/44f7ca59496a46f8ba4507e482235eed
I20260810 23:29:13.983825 8369 log.cc:1196] T 44f7ca59496a46f8ba4507e482235eed P 2f41eb4f81424f35b0f19a66421a8cd0: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/wal/wals/44f7ca59496a46f8ba4507e482235eed
I20260810 23:29:13.984385 8369 ts_tablet_manager.cc:1939] T 44f7ca59496a46f8ba4507e482235eed P 2f41eb4f81424f35b0f19a66421a8cd0: Deleting consensus metadata
I20260810 23:29:13.984848 8367 ts_tablet_manager.cc:1939] T 44f7ca59496a46f8ba4507e482235eed P 4875e7ba636a4c9fa099a90d2abb0e7c: Deleting consensus metadata
I20260810 23:29:13.986560 7521 catalog_manager.cc:4841] TS 2f41eb4f81424f35b0f19a66421a8cd0 (127.14.175.67:36617): tablet 44f7ca59496a46f8ba4507e482235eed (table test-table [id=cda23cf97ca64034ad2ff14a43793d9d]) successfully deleted
I20260810 23:29:13.987072 7522 catalog_manager.cc:4841] TS 4875e7ba636a4c9fa099a90d2abb0e7c (127.14.175.65:43593): tablet 44f7ca59496a46f8ba4507e482235eed (table test-table [id=cda23cf97ca64034ad2ff14a43793d9d]) successfully deleted
I20260810 23:29:13.993511 8323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:13.954664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41878 (local address 127.14.175.65:43593)
0810 23:29:13.954853 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 23:29:13.954859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:29:13.969969 (+ 15110us) server_negotiation.cc:407] Connection header received
0810 23:29:13.970182 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:29:13.970187 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:29:13.970293 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:29:13.970405 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:29:13.972654 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:13.973940 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:13.978233 (+ 4293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:13.978928 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:13.979033 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:13.985791 (+ 6758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:29:13.985811 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:29:13.985832 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:29:13.985880 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:29:13.990246 (+ 4366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:13.991205 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:13.991213 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:13.991215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:13.991360 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:13.991787 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:13.991791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:13.991793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:13.992234 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:29:13.992538 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0810 23:29:13.992880 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 23:29:13.993113 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 23:29:13.994305 7861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:29:14.131374 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 7651
I20260810 23:29:14.161028 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 7876
Aug 10 23:29:14 dist-test-slave-rmhg krb5kdc[7469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.66@KRBTEST.COM
I20260810 23:29:14.177610 8324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:14.162068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56454 (local address 127.14.175.66:37175)
0810 23:29:14.162276 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 23:29:14.162282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:29:14.162306 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:29:14.162411 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:29:14.162416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:29:14.162484 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:29:14.162565 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:29:14.164031 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:14.164793 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:14.166221 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:14.166635 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:14.166759 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:14.171408 (+ 4649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:29:14.171428 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:29:14.171445 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:29:14.171493 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:29:14.174401 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:14.175207 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:14.175216 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:14.175218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:14.175373 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:14.175795 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:14.175800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:14.175802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:14.176246 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:29:14.176550 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0810 23:29:14.176915 (+ 365us) server_negotiation.cc:299] Negotiation successful
0810 23:29:14.177308 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:29:14.178332 8073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:29:14.268782 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 7876
I20260810 23:29:14.300454 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 8089
Aug 10 23:29:14 dist-test-slave-rmhg krb5kdc[7469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.67@KRBTEST.COM
I20260810 23:29:14.325783 8330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:14.301319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51716 (local address 127.14.175.67:36617)
0810 23:29:14.301505 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 23:29:14.301511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:29:14.301594 (+ 83us) server_negotiation.cc:407] Connection header received
0810 23:29:14.301873 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:29:14.301880 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:29:14.302022 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:29:14.302132 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:29:14.305300 (+ 3168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:14.306421 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:14.308279 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:14.308862 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:14.308995 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:14.315217 (+ 6222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:29:14.315238 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:29:14.315263 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:29:14.315315 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:29:14.321944 (+ 6629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:14.322793 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:14.322804 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:14.322808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:14.322992 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:14.323446 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:14.323453 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:14.323456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:14.323995 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:29:14.324347 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0810 23:29:14.325036 (+ 689us) server_negotiation.cc:299] Negotiation successful
0810 23:29:14.325349 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 23:29:14.326686 8292 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:29:14.457122 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 8089
I20260810 23:29:14.483179 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 7489
I20260810 23:29:14.484593 7602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:29:14.599233 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 7489
I20260810 23:29:34.629266 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:45843
--webserver_interface=127.14.175.126
--webserver_port=34573
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:42347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:45843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:29:34.991266 8380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:29:34.991600 8380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:29:34.991703 8380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:29:34.999250 8380 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 23:29:34.999365 8380 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 23:29:34.999403 8380 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 23:29:34.999454 8380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:29:34.999521 8380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:29:34.999594 8380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:29:34.999639 8380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:29:34.999691 8380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:29:35.010239 8380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:42347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:45843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:45843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=34573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:29:35.011580 8380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:29:35.012948 8380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:29:35.040161 8395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:29:35.040293 8392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:29:35.041174 8390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:29:35.043793 8380 server_base.cc:1034] running on GCE node
I20260810 23:29:35.044878 8380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:29:35.047052 8380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:29:35.048317 8380 hybrid_clock.cc:648] HybridClock initialized: now 1786404575048235 us; error 53 us; skew 500 ppm
I20260810 23:29:35.048714 8380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:29:35 dist-test-slave-rmhg krb5kdc[7469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404575, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:29:35.052791 8380 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:29:35.054744 8380 webserver.cc:460] Webserver started at http://127.14.175.126:34573/ using document root <none> and password file <none>
I20260810 23:29:35.055588 8380 fs_manager.cc:359] Metadata directory not provided
I20260810 23:29:35.055711 8380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:29:35.062578 8380 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:29:35.065932 8404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:29:35.067543 8380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:29:35.067724 8380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "3f3e6979d111499fb5b7805eabf0241e"
format_stamp: "Formatted at 2026-08-10 23:29:11 on dist-test-slave-rmhg"
server_key: "a1129d012a7ae976ff06d2ec6dde2c51"
server_key_iv: "6d6bf709d15e134d04e2b1560dcab157"
server_key_version: "encryptionkey@0"
I20260810 23:29:35.068840 8380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:29:35.089432 8380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:29:35.094585 8380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:29:35.094880 8380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:29:35.115938 8380 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:45843
I20260810 23:29:35.115958 8484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:45843 every 8 connection(s)
I20260810 23:29:35.117856 8380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:29:35.121639 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 8380
I20260810 23:29:35.122507 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.65:43593
--local_ip_for_outbound_sockets=127.14.175.65
--tserver_master_addrs=127.14.175.126:45843
--webserver_port=46523
--webserver_interface=127.14.175.65
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:42347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:29:35.126736 8485 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:29:35.131506 8485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e: Bootstrap starting.
I20260810 23:29:35.153667 8485 log.cc:824] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:29:35.167336 8485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:29:35.167948 8485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e: Bootstrap complete.
I20260810 23:29:35.178527 8485 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3e6979d111499fb5b7805eabf0241e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 45843 } }
I20260810 23:29:35.179497 8485 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f3e6979d111499fb5b7805eabf0241e, State: Initialized, Role: FOLLOWER
I20260810 23:29:35.180159 8485 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3e6979d111499fb5b7805eabf0241e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 45843 } }
I20260810 23:29:35.180420 8485 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:29:35.180547 8485 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:29:35.180724 8485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:29:35.185341 8485 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3e6979d111499fb5b7805eabf0241e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 45843 } }
I20260810 23:29:35.185926 8485 leader_election.cc:302] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f3e6979d111499fb5b7805eabf0241e; no voters:
I20260810 23:29:35.186622 8485 leader_election.cc:288] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 23:29:35.186776 8489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [term 2 FOLLOWER]: Leader election won for term 2
I20260810 23:29:35.187886 8489 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [term 2 LEADER]: Becoming Leader. State: Replica: 3f3e6979d111499fb5b7805eabf0241e, State: Running, Role: LEADER
I20260810 23:29:35.188598 8489 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3e6979d111499fb5b7805eabf0241e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 45843 } }
I20260810 23:29:35.189071 8485 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:29:35.191438 8497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f3e6979d111499fb5b7805eabf0241e. Latest consensus state: current_term: 2 leader_uuid: "3f3e6979d111499fb5b7805eabf0241e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3e6979d111499fb5b7805eabf0241e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 45843 } } }
I20260810 23:29:35.191469 8495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3f3e6979d111499fb5b7805eabf0241e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3e6979d111499fb5b7805eabf0241e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 45843 } } }
I20260810 23:29:35.191870 8497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [sys.catalog]: This master's current role is: LEADER
I20260810 23:29:35.191891 8495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [sys.catalog]: This master's current role is: LEADER
I20260810 23:29:35.202639 8506 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:29:35.212494 8506 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:29:35.214483 8506 catalog_manager.cc:1259] Loaded cluster ID: 506ab82d48304dcd94becd57e18afe2a
I20260810 23:29:35.214717 8506 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:29:35.220188 8506 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:29:35.222313 8506 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e: Loaded TSK: 0
I20260810 23:29:35.224797 8524 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 23:29:35.252626 8506 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e: Generated new TSK 1
I20260810 23:29:35.253059 8506 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 23:29:35.540897 8487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:29:35.541277 8487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:29:35.541453 8487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:29:35.549017 8487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:29:35.549140 8487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:29:35.549317 8487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.65
I20260810 23:29:35.559455 8487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:42347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.65:43593
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.175.65
--webserver_port=46523
--tserver_master_addrs=127.14.175.126:45843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.65
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:29:35.560832 8487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:29:35.562276 8487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:29:35.577775 8537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:29:35.577813 8535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:29:35.578570 8541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:29:35.579420 8487 server_base.cc:1034] running on GCE node
I20260810 23:29:35.580230 8487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:29:35.581489 8487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:29:35.582746 8487 hybrid_clock.cc:648] HybridClock initialized: now 1786404575582677 us; error 57 us; skew 500 ppm
I20260810 23:29:35.583189 8487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:29:35 dist-test-slave-rmhg krb5kdc[7469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404575, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:29:35.587265 8487 init.cc:377] Logged in from keytab as kudu/127.14.175.65@KRBTEST.COM (short username kudu)
I20260810 23:29:35.589247 8487 webserver.cc:460] Webserver started at http://127.14.175.65:46523/ using document root <none> and password file <none>
I20260810 23:29:35.590269 8487 fs_manager.cc:359] Metadata directory not provided
I20260810 23:29:35.590442 8487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:29:35.598065 8487 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260810 23:29:35.601763 8554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:29:35.603526 8487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:29:35.603729 8487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/wal
uuid: "4875e7ba636a4c9fa099a90d2abb0e7c"
format_stamp: "Formatted at 2026-08-10 23:29:12 on dist-test-slave-rmhg"
server_key: "cc6fb69dac1033ba3e7baba662a4ba6e"
server_key_iv: "03536a959811b9e3f943b60185fbc337"
server_key_version: "encryptionkey@0"
I20260810 23:29:35.605130 8487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:29:35.647548 8487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:29:35.651885 8487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:29:35.652122 8487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:29:35.653097 8487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:29:35.654575 8487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:29:35.654693 8487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:29:35.654785 8487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:29:35.654841 8487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:29:35.685704 8487 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.65:43593
I20260810 23:29:35.685824 8742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.65:43593 every 8 connection(s)
I20260810 23:29:35.687207 8487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
I20260810 23:29:35.695794 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 8487
I20260810 23:29:35.696434 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.66:37175
--local_ip_for_outbound_sockets=127.14.175.66
--tserver_master_addrs=127.14.175.126:45843
--webserver_port=36815
--webserver_interface=127.14.175.66
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:42347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:29:35 dist-test-slave-rmhg krb5kdc[7469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404575, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:29:35.711962 8751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:35.690626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.65:44455)
0810 23:29:35.691473 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:35.691525 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 23:29:35.693349 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:29:35.694764 (+ 1415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:29:35.694772 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:29:35.695336 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:29:35.695945 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:35.695972 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:35.697754 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:35.697758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:35.699064 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:35.699078 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:35.700165 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:35.700173 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:35.700456 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:35.701123 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:29:35.701177 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:29:35.703500 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:29:35.707123 (+ 3623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:29:35.707133 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:29:35.707150 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:29:35.707639 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:29:35.708132 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:29:35.708138 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:29:35.708142 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:29:35.708353 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:29:35.709186 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:29:35.709196 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:29:35.709573 (+ 377us) client_negotiation.cc:769] Sending connection context
0810 23:29:35.709848 (+ 275us) client_negotiation.cc:240] Negotiation successful
0810 23:29:35.710514 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":347,"threads_started":1}
I20260810 23:29:35.711962 8750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:35.690219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:44455 (local address 127.14.175.126:45843)
0810 23:29:35.690901 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0810 23:29:35.690920 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 23:29:35.691699 (+ 779us) server_negotiation.cc:407] Connection header received
0810 23:29:35.693868 (+ 2169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:29:35.693882 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:29:35.694242 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:29:35.694551 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:29:35.696186 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:35.697578 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:35.699227 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:35.699898 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:35.700107 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:35.703820 (+ 3713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:29:35.703829 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:29:35.703846 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:29:35.703888 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:29:35.706939 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:35.707794 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:35.707803 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:35.707810 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:35.708003 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:35.708470 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:35.708474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:35.708476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:35.709004 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:29:35.709245 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0810 23:29:35.710109 (+ 864us) server_negotiation.cc:299] Negotiation successful
0810 23:29:35.710577 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":255,"threads_started":1}
I20260810 23:29:35.715080 8743 heartbeater.cc:347] Connected to a master server at 127.14.175.126:45843
I20260810 23:29:35.715452 8743 heartbeater.cc:464] Registering TS with master...
I20260810 23:29:35.716444 8743 heartbeater.cc:511] Master 127.14.175.126:45843 requested a full tablet report, sending...
I20260810 23:29:35.718786 8425 ts_manager.cc:194] Registered new tserver with Master: 4875e7ba636a4c9fa099a90d2abb0e7c (127.14.175.65:43593)
I20260810 23:29:35.720304 8425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.65@KRBTEST.COM'} at 127.14.175.65:44455
W20260810 23:29:36.048982 8752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:29:36.049311 8752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:29:36.049456 8752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:29:36.056993 8752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:29:36.057116 8752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:29:36.057283 8752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.66
I20260810 23:29:36.067528 8752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:42347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.66:37175
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.175.66
--webserver_port=36815
--tserver_master_addrs=127.14.175.126:45843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.66
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:29:36.068982 8752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:29:36.070418 8752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:29:36.084409 8764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:29:36.084496 8761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:29:36.084524 8762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:29:36.085430 8752 server_base.cc:1034] running on GCE node
I20260810 23:29:36.086054 8752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:29:36.087052 8752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:29:36.088280 8752 hybrid_clock.cc:648] HybridClock initialized: now 1786404576088220 us; error 51 us; skew 500 ppm
I20260810 23:29:36.088614 8752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:29:36 dist-test-slave-rmhg krb5kdc[7469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404576, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:29:36.091917 8752 init.cc:377] Logged in from keytab as kudu/127.14.175.66@KRBTEST.COM (short username kudu)
I20260810 23:29:36.093448 8752 webserver.cc:460] Webserver started at http://127.14.175.66:36815/ using document root <none> and password file <none>
I20260810 23:29:36.094329 8752 fs_manager.cc:359] Metadata directory not provided
I20260810 23:29:36.094511 8752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:29:36.100399 8752 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260810 23:29:36.103443 8774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:29:36.104914 8752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:29:36.105109 8752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/wal
uuid: "dc053cfca0964ebf87c75bd4dee18bc7"
format_stamp: "Formatted at 2026-08-10 23:29:12 on dist-test-slave-rmhg"
server_key: "d2aa37f0ba78c8eda5f8708686eadfa9"
server_key_iv: "099366519a0caba015482782ac9b470a"
server_key_version: "encryptionkey@0"
I20260810 23:29:36.106186 8752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:29:36.131461 8752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:29:36.136225 8752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:29:36.136504 8752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:29:36.137630 8752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:29:36.139101 8752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:29:36.139209 8752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:29:36.139322 8752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:29:36.139393 8752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:29:36.170861 8752 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.66:37175
I20260810 23:29:36.170899 8963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.66:37175 every 8 connection(s)
I20260810 23:29:36.172442 8752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
I20260810 23:29:36.178378 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 8752
I20260810 23:29:36.179018 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.67:36617
--local_ip_for_outbound_sockets=127.14.175.67
--tserver_master_addrs=127.14.175.126:45843
--webserver_port=37915
--webserver_interface=127.14.175.67
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:42347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:29:36 dist-test-slave-rmhg krb5kdc[7469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404576, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:29:36.198630 8750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:36.175303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:34293 (local address 127.14.175.126:45843)
0810 23:29:36.175495 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 23:29:36.175499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:29:36.176852 (+ 1353us) server_negotiation.cc:407] Connection header received
0810 23:29:36.178966 (+ 2114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:29:36.178972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:29:36.179113 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:29:36.179237 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:29:36.181036 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:36.182107 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:36.184483 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:36.185080 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:36.185212 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:36.192498 (+ 7286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:29:36.192504 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:29:36.192508 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:29:36.192548 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:29:36.195213 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:36.196078 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:36.196083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:36.196086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:36.196240 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:36.196702 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:36.196707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:36.196709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:36.197024 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:29:36.197193 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 23:29:36.198003 (+ 810us) server_negotiation.cc:299] Negotiation successful
0810 23:29:36.198279 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 23:29:36.199740 8967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:36.175784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.66:34293)
0810 23:29:36.176540 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:36.176695 (+ 155us) client_negotiation.cc:174] Beginning negotiation
0810 23:29:36.178636 (+ 1941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:29:36.179584 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:29:36.179594 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:29:36.180174 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:29:36.180828 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:36.180849 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:36.182332 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:36.182337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:36.184261 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:36.184282 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:36.185595 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:36.185602 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:36.185926 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:36.189176 (+ 3250us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:29:36.189215 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:29:36.192187 (+ 2972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:29:36.195401 (+ 3214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:29:36.195413 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:29:36.195432 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:29:36.195935 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:29:36.196369 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:29:36.196373 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:29:36.196377 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:29:36.196581 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:29:36.197195 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:29:36.197205 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:29:36.197587 (+ 382us) client_negotiation.cc:769] Sending connection context
0810 23:29:36.197896 (+ 309us) client_negotiation.cc:240] Negotiation successful
0810 23:29:36.198509 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":292,"threads_started":1}
I20260810 23:29:36.201395 8965 heartbeater.cc:347] Connected to a master server at 127.14.175.126:45843
I20260810 23:29:36.201814 8965 heartbeater.cc:464] Registering TS with master...
I20260810 23:29:36.202819 8965 heartbeater.cc:511] Master 127.14.175.126:45843 requested a full tablet report, sending...
I20260810 23:29:36.204783 8425 ts_manager.cc:194] Registered new tserver with Master: dc053cfca0964ebf87c75bd4dee18bc7 (127.14.175.66:37175)
I20260810 23:29:36.206068 8425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.66@KRBTEST.COM'} at 127.14.175.66:34293
W20260810 23:29:36.539243 8968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:29:36.539592 8968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:29:36.539749 8968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:29:36.547344 8968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:29:36.547487 8968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:29:36.547616 8968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.67
I20260810 23:29:36.558065 8968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:42347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.67:36617
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.175.67
--webserver_port=37915
--tserver_master_addrs=127.14.175.126:45843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.67
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:29:36.559425 8968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:29:36.560833 8968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:29:36.575095 8982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:29:36.575033 8984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:29:36.575124 8977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:29:36.576115 8968 server_base.cc:1034] running on GCE node
I20260810 23:29:36.576838 8968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:29:36.577991 8968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:29:36.579216 8968 hybrid_clock.cc:648] HybridClock initialized: now 1786404576579150 us; error 69 us; skew 500 ppm
I20260810 23:29:36.579561 8968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:29:36 dist-test-slave-rmhg krb5kdc[7469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404576, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:29:36.583305 8968 init.cc:377] Logged in from keytab as kudu/127.14.175.67@KRBTEST.COM (short username kudu)
I20260810 23:29:36.585013 8968 webserver.cc:460] Webserver started at http://127.14.175.67:37915/ using document root <none> and password file <none>
I20260810 23:29:36.585906 8968 fs_manager.cc:359] Metadata directory not provided
I20260810 23:29:36.586058 8968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:29:36.592128 8968 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:29:36.595258 8996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:29:36.596848 8968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:29:36.597045 8968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/wal
uuid: "2f41eb4f81424f35b0f19a66421a8cd0"
format_stamp: "Formatted at 2026-08-10 23:29:13 on dist-test-slave-rmhg"
server_key: "dde0da0c43fcf4d8c1a7178d0c4bd29f"
server_key_iv: "9d9e62a9d4195e91536755027ee856ff"
server_key_version: "encryptionkey@0"
I20260810 23:29:36.598258 8968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:29:36.627112 8968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:29:36.631979 8968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:29:36.632279 8968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:29:36.633404 8968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:29:36.634959 8968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:29:36.635068 8968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:29:36.635187 8968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:29:36.635258 8968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:29:36.667824 8968 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.67:36617
I20260810 23:29:36.667900 9181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.67:36617 every 8 connection(s)
I20260810 23:29:36.669423 8968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
I20260810 23:29:36.672607 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 8968
Aug 10 23:29:36 dist-test-slave-rmhg krb5kdc[7469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404576, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:29:36.693490 8750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:36.672193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:43001 (local address 127.14.175.126:45843)
0810 23:29:36.672414 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 23:29:36.672421 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:29:36.673781 (+ 1360us) server_negotiation.cc:407] Connection header received
0810 23:29:36.675953 (+ 2172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:29:36.675958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:29:36.676046 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:29:36.676158 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:29:36.677882 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:36.678987 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:36.681049 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:36.681717 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:36.681849 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:36.685986 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:29:36.685994 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:29:36.685999 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:29:36.686056 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:29:36.690077 (+ 4021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:36.691086 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:36.691092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:36.691096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:36.691248 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:36.691663 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:36.691668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:36.691670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:36.691969 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:29:36.692106 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 23:29:36.692850 (+ 744us) server_negotiation.cc:299] Negotiation successful
0810 23:29:36.693101 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 23:29:36.693723 9185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:36.673507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39782 (local address 127.14.175.126:45843)
0810 23:29:36.674424 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0810 23:29:36.674430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:29:36.677758 (+ 3328us) server_negotiation.cc:407] Connection header received
0810 23:29:36.677908 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:29:36.677911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:29:36.677975 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:29:36.678059 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:29:36.680016 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:36.681112 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:36.682917 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:36.683630 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:36.683758 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:36.687019 (+ 3261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:29:36.687026 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:29:36.687031 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:29:36.687090 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:29:36.690612 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:36.691557 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:36.691562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:36.691566 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:36.691693 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:36.692078 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:36.692083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:36.692086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:36.692362 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:29:36.692610 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0810 23:29:36.692939 (+ 329us) server_negotiation.cc:299] Negotiation successful
0810 23:29:36.693293 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":806,"thread_start_us":674,"threads_started":1}
I20260810 23:29:36.694705 9184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:36.672720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.67:43001)
0810 23:29:36.673619 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:36.673668 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 23:29:36.675658 (+ 1990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:29:36.676430 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:29:36.676439 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:29:36.677000 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:29:36.677672 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:36.677693 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:36.679174 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:36.679179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:36.680894 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:36.680910 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:36.681925 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:36.681930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:36.682218 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:36.682881 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:29:36.682915 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:29:36.685714 (+ 2799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:29:36.690325 (+ 4611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:29:36.690337 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:29:36.690354 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:29:36.690935 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:29:36.691346 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:29:36.691349 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:29:36.691351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:29:36.691531 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:29:36.692130 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:29:36.692140 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:29:36.692480 (+ 340us) client_negotiation.cc:769] Sending connection context
0810 23:29:36.692722 (+ 242us) client_negotiation.cc:240] Negotiation successful
0810 23:29:36.693319 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":360,"threads_started":1}
I20260810 23:29:36.696206 9182 heartbeater.cc:347] Connected to a master server at 127.14.175.126:45843
I20260810 23:29:36.696511 9182 heartbeater.cc:464] Registering TS with master...
I20260810 23:29:36.697296 9182 heartbeater.cc:511] Master 127.14.175.126:45843 requested a full tablet report, sending...
I20260810 23:29:36.698827 8423 ts_manager.cc:194] Registered new tserver with Master: 2f41eb4f81424f35b0f19a66421a8cd0 (127.14.175.67:36617)
I20260810 23:29:36.699750 8423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.67@KRBTEST.COM'} at 127.14.175.67:43001
I20260810 23:29:36.709604 15037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:29:36.717326 9185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:36.710770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39786 (local address 127.14.175.126:45843)
0810 23:29:36.710963 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 23:29:36.710969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:29:36.711482 (+ 513us) server_negotiation.cc:407] Connection header received
0810 23:29:36.711754 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:29:36.711758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:29:36.711914 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:29:36.712071 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:29:36.712586 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:36.713362 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:36.714937 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:36.715417 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:36.715534 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:36.716127 (+ 593us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:29:36.716432 (+ 305us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 23:29:36.716959 (+ 527us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39786: token expired
Metrics: {"server-negotiator.queue_time_us":46}
W20260810 23:29:36.717451 9185 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39786: token expired
W20260810 23:29:36.717933 8309 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.14.175.126:45843)
I20260810 23:29:36.724284 8743 heartbeater.cc:503] Master 127.14.175.126:45843 was elected leader, sending a full tablet report...
I20260810 23:29:36.732131 9185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:36.718606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39798 (local address 127.14.175.126:45843)
0810 23:29:36.718781 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 23:29:36.718787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:29:36.718805 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:29:36.718855 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:29:36.718859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:29:36.718919 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:29:36.719010 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:29:36.720452 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:36.721268 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:36.722740 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:36.723182 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:36.723296 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:36.726756 (+ 3460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:29:36.726761 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:29:36.726765 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:29:36.726800 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:29:36.729510 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:36.730260 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:36.730265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:36.730267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:36.730438 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:36.730823 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:36.730827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:36.730830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:36.731200 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:29:36.731402 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0810 23:29:36.731647 (+ 245us) server_negotiation.cc:299] Negotiation successful
0810 23:29:36.731857 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 23:29:36.735936 8423 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:29:36.743562 8423 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:29:36.780643 9199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:36.756691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41728 (local address 127.14.175.66:37175)
0810 23:29:36.771044 (+ 14353us) server_negotiation.cc:206] Beginning negotiation
0810 23:29:36.771053 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:29:36.771085 (+ 32us) server_negotiation.cc:407] Connection header received
0810 23:29:36.771199 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:29:36.771211 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:29:36.771511 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:29:36.772499 (+ 988us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:29:36.773103 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:36.774692 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:36.777872 (+ 3180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:36.779377 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:36.779505 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:36.779726 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0810 23:29:36.780124 (+ 398us) server_negotiation.cc:299] Negotiation successful
0810 23:29:36.780308 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14167,"thread_start_us":14078,"threads_started":1}
I20260810 23:29:36.780725 9201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:36.756438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:37175 (local address 127.0.0.1:41728)
0810 23:29:36.767389 (+ 10951us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:36.767478 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0810 23:29:36.767755 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:29:36.771793 (+ 4038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:29:36.771800 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:29:36.771839 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:29:36.772807 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:36.772823 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:36.774866 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:36.774871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:36.777499 (+ 2628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:36.777524 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:36.779627 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:36.779634 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:36.779876 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:36.779923 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 23:29:36.780044 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 23:29:36.780308 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10571,"mutex_wait_us":35,"thread_start_us":14836,"threads_started":1}
I20260810 23:29:36.780810 9203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:36.757551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43632 (local address 127.14.175.65:43593)
0810 23:29:36.772356 (+ 14805us) server_negotiation.cc:206] Beginning negotiation
0810 23:29:36.772365 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:29:36.772394 (+ 29us) server_negotiation.cc:407] Connection header received
0810 23:29:36.772497 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:29:36.772504 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:29:36.772728 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:29:36.772936 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:29:36.773745 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:36.775373 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:36.777736 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:36.779113 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:36.779272 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:36.779550 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0810 23:29:36.780277 (+ 727us) server_negotiation.cc:299] Negotiation successful
0810 23:29:36.780456 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14562,"thread_start_us":14408,"threads_started":1}
I20260810 23:29:36.781720 9198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:36.756862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:43593 (local address 127.0.0.1:43632)
0810 23:29:36.771562 (+ 14700us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:36.771583 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:29:36.771706 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:29:36.772949 (+ 1243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:29:36.772954 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:29:36.772985 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:29:36.773425 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:36.773487 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:36.775558 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:36.775566 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:36.777523 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:36.777543 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:36.779714 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:36.779721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:36.779923 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:36.779959 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 23:29:36.780043 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 23:29:36.781286 (+ 1243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14556,"mutex_wait_us":1017,"thread_start_us":11169,"threads_started":1}
I20260810 23:29:36.782430 9202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:36.756806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:36617 (local address 127.0.0.1:52572)
0810 23:29:36.770363 (+ 13557us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:36.770385 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:29:36.770517 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:29:36.772522 (+ 2005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:29:36.772527 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:29:36.772553 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:29:36.772987 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:36.772999 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:36.775500 (+ 2501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:36.775506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:36.779083 (+ 3577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:36.779103 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:36.781250 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:36.781256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:36.781432 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:36.781471 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 23:29:36.781563 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 23:29:36.781725 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13414,"mutex_wait_us":52,"thread_start_us":13263,"threads_started":1}
I20260810 23:29:36.783349 9204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:36.758070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52572 (local address 127.14.175.67:36617)
0810 23:29:36.771890 (+ 13820us) server_negotiation.cc:206] Beginning negotiation
0810 23:29:36.771900 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:29:36.771940 (+ 40us) server_negotiation.cc:407] Connection header received
0810 23:29:36.772044 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:29:36.772051 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:29:36.772291 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:29:36.773239 (+ 948us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:29:36.773604 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:36.775332 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:36.779299 (+ 3967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:36.780838 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:36.780981 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:36.782034 (+ 1053us) server_negotiation.cc:1035] Waiting for connection context
0810 23:29:36.782420 (+ 386us) server_negotiation.cc:299] Negotiation successful
0810 23:29:36.782991 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13636,"thread_start_us":13460,"threads_started":1}
I20260810 23:29:36.785501 8861 tablet_service.cc:1467] Processing CreateTablet for tablet 4c96bf37396a4a689a695bf4382fe0a3 (DEFAULT_TABLE table=test-table [id=ee92fc1edb8f41be85820c618bbdcf54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:29:36.787091 8861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c96bf37396a4a689a695bf4382fe0a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:29:36.787184 9073 tablet_service.cc:1467] Processing CreateTablet for tablet 4c96bf37396a4a689a695bf4382fe0a3 (DEFAULT_TABLE table=test-table [id=ee92fc1edb8f41be85820c618bbdcf54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:29:36.788630 9073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c96bf37396a4a689a695bf4382fe0a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:29:36.789265 8639 tablet_service.cc:1467] Processing CreateTablet for tablet 4c96bf37396a4a689a695bf4382fe0a3 (DEFAULT_TABLE table=test-table [id=ee92fc1edb8f41be85820c618bbdcf54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:29:36.791162 8639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c96bf37396a4a689a695bf4382fe0a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:29:36.801892 9205 tablet_bootstrap.cc:492] T 4c96bf37396a4a689a695bf4382fe0a3 P 2f41eb4f81424f35b0f19a66421a8cd0: Bootstrap starting.
I20260810 23:29:36.805975 9206 tablet_bootstrap.cc:492] T 4c96bf37396a4a689a695bf4382fe0a3 P dc053cfca0964ebf87c75bd4dee18bc7: Bootstrap starting.
I20260810 23:29:36.807124 9205 tablet_bootstrap.cc:654] T 4c96bf37396a4a689a695bf4382fe0a3 P 2f41eb4f81424f35b0f19a66421a8cd0: Neither blocks nor log segments found. Creating new log.
I20260810 23:29:36.808390 9209 tablet_bootstrap.cc:492] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c: Bootstrap starting.
I20260810 23:29:36.808568 9205 log.cc:824] T 4c96bf37396a4a689a695bf4382fe0a3 P 2f41eb4f81424f35b0f19a66421a8cd0: Log is configured to *not* fsync() on all Append() calls
I20260810 23:29:36.810000 9206 tablet_bootstrap.cc:654] T 4c96bf37396a4a689a695bf4382fe0a3 P dc053cfca0964ebf87c75bd4dee18bc7: Neither blocks nor log segments found. Creating new log.
I20260810 23:29:36.811323 9206 log.cc:824] T 4c96bf37396a4a689a695bf4382fe0a3 P dc053cfca0964ebf87c75bd4dee18bc7: Log is configured to *not* fsync() on all Append() calls
I20260810 23:29:36.812155 9209 tablet_bootstrap.cc:654] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c: Neither blocks nor log segments found. Creating new log.
I20260810 23:29:36.812697 9205 tablet_bootstrap.cc:492] T 4c96bf37396a4a689a695bf4382fe0a3 P 2f41eb4f81424f35b0f19a66421a8cd0: No bootstrap required, opened a new log
I20260810 23:29:36.813092 9205 ts_tablet_manager.cc:1397] T 4c96bf37396a4a689a695bf4382fe0a3 P 2f41eb4f81424f35b0f19a66421a8cd0: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260810 23:29:36.813553 9209 log.cc:824] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c: Log is configured to *not* fsync() on all Append() calls
I20260810 23:29:36.814790 9206 tablet_bootstrap.cc:492] T 4c96bf37396a4a689a695bf4382fe0a3 P dc053cfca0964ebf87c75bd4dee18bc7: No bootstrap required, opened a new log
I20260810 23:29:36.815088 9206 ts_tablet_manager.cc:1397] T 4c96bf37396a4a689a695bf4382fe0a3 P dc053cfca0964ebf87c75bd4dee18bc7: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260810 23:29:36.817288 9209 tablet_bootstrap.cc:492] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c: No bootstrap required, opened a new log
I20260810 23:29:36.817603 9209 ts_tablet_manager.cc:1397] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260810 23:29:36.821285 9205 raft_consensus.cc:348] T 4c96bf37396a4a689a695bf4382fe0a3 P 2f41eb4f81424f35b0f19a66421a8cd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc053cfca0964ebf87c75bd4dee18bc7" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 37175 } } peers { permanent_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 36617 } } peers { permanent_uuid: "4875e7ba636a4c9fa099a90d2abb0e7c" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 43593 } }
I20260810 23:29:36.821687 9205 raft_consensus.cc:374] T 4c96bf37396a4a689a695bf4382fe0a3 P 2f41eb4f81424f35b0f19a66421a8cd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:29:36.821805 9205 raft_consensus.cc:729] T 4c96bf37396a4a689a695bf4382fe0a3 P 2f41eb4f81424f35b0f19a66421a8cd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f41eb4f81424f35b0f19a66421a8cd0, State: Initialized, Role: FOLLOWER
I20260810 23:29:36.822450 9205 consensus_queue.cc:260] T 4c96bf37396a4a689a695bf4382fe0a3 P 2f41eb4f81424f35b0f19a66421a8cd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc053cfca0964ebf87c75bd4dee18bc7" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 37175 } } peers { permanent_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 36617 } } peers { permanent_uuid: "4875e7ba636a4c9fa099a90d2abb0e7c" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 43593 } }
I20260810 23:29:36.824321 9205 ts_tablet_manager.cc:1428] T 4c96bf37396a4a689a695bf4382fe0a3 P 2f41eb4f81424f35b0f19a66421a8cd0: Time spent starting tablet: real 0.011s user 0.004s sys 0.005s
I20260810 23:29:36.824983 9182 heartbeater.cc:503] Master 127.14.175.126:45843 was elected leader, sending a full tablet report...
I20260810 23:29:36.825861 9206 raft_consensus.cc:348] T 4c96bf37396a4a689a695bf4382fe0a3 P dc053cfca0964ebf87c75bd4dee18bc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc053cfca0964ebf87c75bd4dee18bc7" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 37175 } } peers { permanent_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 36617 } } peers { permanent_uuid: "4875e7ba636a4c9fa099a90d2abb0e7c" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 43593 } }
I20260810 23:29:36.826362 9206 raft_consensus.cc:374] T 4c96bf37396a4a689a695bf4382fe0a3 P dc053cfca0964ebf87c75bd4dee18bc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:29:36.826470 9206 raft_consensus.cc:729] T 4c96bf37396a4a689a695bf4382fe0a3 P dc053cfca0964ebf87c75bd4dee18bc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc053cfca0964ebf87c75bd4dee18bc7, State: Initialized, Role: FOLLOWER
I20260810 23:29:36.827162 9206 consensus_queue.cc:260] T 4c96bf37396a4a689a695bf4382fe0a3 P dc053cfca0964ebf87c75bd4dee18bc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc053cfca0964ebf87c75bd4dee18bc7" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 37175 } } peers { permanent_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 36617 } } peers { permanent_uuid: "4875e7ba636a4c9fa099a90d2abb0e7c" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 43593 } }
I20260810 23:29:36.828439 8965 heartbeater.cc:503] Master 127.14.175.126:45843 was elected leader, sending a full tablet report...
I20260810 23:29:36.828323 9209 raft_consensus.cc:348] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc053cfca0964ebf87c75bd4dee18bc7" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 37175 } } peers { permanent_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 36617 } } peers { permanent_uuid: "4875e7ba636a4c9fa099a90d2abb0e7c" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 43593 } }
I20260810 23:29:36.828802 9209 raft_consensus.cc:374] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:29:36.828912 9209 raft_consensus.cc:729] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4875e7ba636a4c9fa099a90d2abb0e7c, State: Initialized, Role: FOLLOWER
I20260810 23:29:36.829288 9206 ts_tablet_manager.cc:1428] T 4c96bf37396a4a689a695bf4382fe0a3 P dc053cfca0964ebf87c75bd4dee18bc7: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260810 23:29:36.829741 9209 consensus_queue.cc:260] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc053cfca0964ebf87c75bd4dee18bc7" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 37175 } } peers { permanent_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 36617 } } peers { permanent_uuid: "4875e7ba636a4c9fa099a90d2abb0e7c" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 43593 } }
I20260810 23:29:36.834506 9209 ts_tablet_manager.cc:1428] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
W20260810 23:29:36.921108 9183 tablet.cc:2369] T 4c96bf37396a4a689a695bf4382fe0a3 P 2f41eb4f81424f35b0f19a66421a8cd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:29:36.924474 8966 tablet.cc:2369] T 4c96bf37396a4a689a695bf4382fe0a3 P dc053cfca0964ebf87c75bd4dee18bc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:29:36.939703 8747 tablet.cc:2369] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:29:36.981197 9216 raft_consensus.cc:482] T 4c96bf37396a4a689a695bf4382fe0a3 P 2f41eb4f81424f35b0f19a66421a8cd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:29:36.981571 9216 raft_consensus.cc:504] T 4c96bf37396a4a689a695bf4382fe0a3 P 2f41eb4f81424f35b0f19a66421a8cd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc053cfca0964ebf87c75bd4dee18bc7" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 37175 } } peers { permanent_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 36617 } } peers { permanent_uuid: "4875e7ba636a4c9fa099a90d2abb0e7c" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 43593 } }
I20260810 23:29:36.983839 9216 leader_election.cc:288] T 4c96bf37396a4a689a695bf4382fe0a3 P 2f41eb4f81424f35b0f19a66421a8cd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc053cfca0964ebf87c75bd4dee18bc7 (127.14.175.66:37175), 4875e7ba636a4c9fa099a90d2abb0e7c (127.14.175.65:43593)
I20260810 23:29:36.991226 9203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:36.984271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:54999 (local address 127.14.175.65:43593)
0810 23:29:36.984515 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 23:29:36.984521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:29:36.984739 (+ 218us) server_negotiation.cc:407] Connection header received
0810 23:29:36.984946 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:29:36.984952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:29:36.985032 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:29:36.985127 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:29:36.985611 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:36.986591 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:36.989011 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:36.990346 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:36.990534 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:36.990617 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 23:29:36.990856 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 23:29:36.991004 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 23:29:36.991360 9184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:36.984205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:43593 (local address 127.14.175.67:54999)
0810 23:29:36.984631 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:36.984648 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:29:36.984802 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:29:36.985197 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:29:36.985202 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:29:36.985235 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:29:36.985468 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:36.985480 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:36.986747 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:36.986751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:36.988782 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:36.988841 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:36.990527 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:36.990534 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:36.990683 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:36.990711 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 23:29:36.990812 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 23:29:36.990978 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":1,"spinlock_wait_cycles":16128}
I20260810 23:29:36.991679 9199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:36.984390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:34709 (local address 127.14.175.66:37175)
0810 23:29:36.985104 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0810 23:29:36.985111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:29:36.985278 (+ 167us) server_negotiation.cc:407] Connection header received
0810 23:29:36.985469 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:29:36.985474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:29:36.985535 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:29:36.985629 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:29:36.986111 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:36.987003 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:36.989355 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:36.990687 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:36.990820 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:36.990888 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 23:29:36.991250 (+ 362us) server_negotiation.cc:299] Negotiation successful
0810 23:29:36.991391 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542}
I20260810 23:29:36.991789 9220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:36.984390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:37175 (local address 127.14.175.67:34709)
0810 23:29:36.985173 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:36.985192 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:29:36.985344 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:29:36.985649 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:29:36.985652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:29:36.985678 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:29:36.985965 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:36.985972 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:36.987141 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:36.987146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:36.989146 (+ 2000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:36.989194 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:36.990888 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:36.990895 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:36.991058 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:36.991091 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 23:29:36.991186 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 23:29:36.991323 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":43,"spinlock_wait_cycles":6272,"thread_start_us":409,"threads_started":1}
I20260810 23:29:36.992492 8666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c96bf37396a4a689a695bf4382fe0a3" candidate_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4875e7ba636a4c9fa099a90d2abb0e7c" is_pre_election: true
I20260810 23:29:36.992741 8897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c96bf37396a4a689a695bf4382fe0a3" candidate_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc053cfca0964ebf87c75bd4dee18bc7" is_pre_election: true
I20260810 23:29:36.993050 8666 raft_consensus.cc:2454] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f41eb4f81424f35b0f19a66421a8cd0 in term 0.
I20260810 23:29:36.993287 8897 raft_consensus.cc:2454] T 4c96bf37396a4a689a695bf4382fe0a3 P dc053cfca0964ebf87c75bd4dee18bc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f41eb4f81424f35b0f19a66421a8cd0 in term 0.
I20260810 23:29:36.993889 9001 leader_election.cc:302] T 4c96bf37396a4a689a695bf4382fe0a3 P 2f41eb4f81424f35b0f19a66421a8cd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f41eb4f81424f35b0f19a66421a8cd0, 4875e7ba636a4c9fa099a90d2abb0e7c; no voters:
I20260810 23:29:36.994360 9216 raft_consensus.cc:2781] T 4c96bf37396a4a689a695bf4382fe0a3 P 2f41eb4f81424f35b0f19a66421a8cd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:29:36.994516 9216 raft_consensus.cc:482] T 4c96bf37396a4a689a695bf4382fe0a3 P 2f41eb4f81424f35b0f19a66421a8cd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:29:36.994632 9216 raft_consensus.cc:3037] T 4c96bf37396a4a689a695bf4382fe0a3 P 2f41eb4f81424f35b0f19a66421a8cd0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:29:36.997040 9216 raft_consensus.cc:504] T 4c96bf37396a4a689a695bf4382fe0a3 P 2f41eb4f81424f35b0f19a66421a8cd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc053cfca0964ebf87c75bd4dee18bc7" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 37175 } } peers { permanent_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 36617 } } peers { permanent_uuid: "4875e7ba636a4c9fa099a90d2abb0e7c" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 43593 } }
I20260810 23:29:36.997777 9216 leader_election.cc:288] T 4c96bf37396a4a689a695bf4382fe0a3 P 2f41eb4f81424f35b0f19a66421a8cd0 [CANDIDATE]: Term 1 election: Requested vote from peers dc053cfca0964ebf87c75bd4dee18bc7 (127.14.175.66:37175), 4875e7ba636a4c9fa099a90d2abb0e7c (127.14.175.65:43593)
I20260810 23:29:36.998456 8666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c96bf37396a4a689a695bf4382fe0a3" candidate_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4875e7ba636a4c9fa099a90d2abb0e7c"
I20260810 23:29:36.998456 8897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c96bf37396a4a689a695bf4382fe0a3" candidate_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc053cfca0964ebf87c75bd4dee18bc7"
I20260810 23:29:36.998724 8897 raft_consensus.cc:3037] T 4c96bf37396a4a689a695bf4382fe0a3 P dc053cfca0964ebf87c75bd4dee18bc7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:29:36.998715 8666 raft_consensus.cc:3037] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:29:37.002132 8897 raft_consensus.cc:2454] T 4c96bf37396a4a689a695bf4382fe0a3 P dc053cfca0964ebf87c75bd4dee18bc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f41eb4f81424f35b0f19a66421a8cd0 in term 1.
I20260810 23:29:37.002162 8666 raft_consensus.cc:2454] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f41eb4f81424f35b0f19a66421a8cd0 in term 1.
I20260810 23:29:37.002774 9001 leader_election.cc:302] T 4c96bf37396a4a689a695bf4382fe0a3 P 2f41eb4f81424f35b0f19a66421a8cd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2f41eb4f81424f35b0f19a66421a8cd0, 4875e7ba636a4c9fa099a90d2abb0e7c, dc053cfca0964ebf87c75bd4dee18bc7; no voters:
I20260810 23:29:37.003188 9216 raft_consensus.cc:2781] T 4c96bf37396a4a689a695bf4382fe0a3 P 2f41eb4f81424f35b0f19a66421a8cd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:29:37.003739 9216 raft_consensus.cc:686] T 4c96bf37396a4a689a695bf4382fe0a3 P 2f41eb4f81424f35b0f19a66421a8cd0 [term 1 LEADER]: Becoming Leader. State: Replica: 2f41eb4f81424f35b0f19a66421a8cd0, State: Running, Role: LEADER
I20260810 23:29:37.004519 9216 consensus_queue.cc:237] T 4c96bf37396a4a689a695bf4382fe0a3 P 2f41eb4f81424f35b0f19a66421a8cd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc053cfca0964ebf87c75bd4dee18bc7" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 37175 } } peers { permanent_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 36617 } } peers { permanent_uuid: "4875e7ba636a4c9fa099a90d2abb0e7c" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 43593 } }
I20260810 23:29:37.010025 8423 catalog_manager.cc:5495] T 4c96bf37396a4a689a695bf4382fe0a3 P 2f41eb4f81424f35b0f19a66421a8cd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f41eb4f81424f35b0f19a66421a8cd0 (127.14.175.67). New cstate: current_term: 1 leader_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc053cfca0964ebf87c75bd4dee18bc7" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 37175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 36617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4875e7ba636a4c9fa099a90d2abb0e7c" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 43593 } health_report { overall_health: UNKNOWN } } }
I20260810 23:29:37.080660 9204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:37.075283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52574 (local address 127.14.175.67:36617)
0810 23:29:37.075493 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 23:29:37.075498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:29:37.075545 (+ 47us) server_negotiation.cc:407] Connection header received
0810 23:29:37.075759 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:29:37.075763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:29:37.075844 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:29:37.075975 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:29:37.076411 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:37.077118 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:37.078639 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:37.079035 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:37.079127 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:37.079581 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:29:37.080121 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:29:37.080305 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0810 23:29:37.080356 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 23:29:37.080444 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 23:29:37.089751 8666 raft_consensus.cc:1261] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [term 1 FOLLOWER]: Refusing update from remote peer 2f41eb4f81424f35b0f19a66421a8cd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:29:37.089751 8897 raft_consensus.cc:1261] T 4c96bf37396a4a689a695bf4382fe0a3 P dc053cfca0964ebf87c75bd4dee18bc7 [term 1 FOLLOWER]: Refusing update from remote peer 2f41eb4f81424f35b0f19a66421a8cd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:29:37.090866 9222 consensus_queue.cc:1035] T 4c96bf37396a4a689a695bf4382fe0a3 P 2f41eb4f81424f35b0f19a66421a8cd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc053cfca0964ebf87c75bd4dee18bc7" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:29:37.091217 9216 consensus_queue.cc:1035] T 4c96bf37396a4a689a695bf4382fe0a3 P 2f41eb4f81424f35b0f19a66421a8cd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4875e7ba636a4c9fa099a90d2abb0e7c" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 43593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:29:37.103157 9225 mvcc.cc:205] Tried to move back new op lower bound from 7317113147747401728 to 7317113147414732800. Current Snapshot: MvccSnapshot[applied={T|T < 7317113147747401728}]
I20260810 23:29:37.117272 8425 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:29:37.117566 8425 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:29:37.121861 8425 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e: Sending DeleteTablet for 3 replicas of tablet 4c96bf37396a4a689a695bf4382fe0a3
I20260810 23:29:37.123245 8861 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c96bf37396a4a689a695bf4382fe0a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:29:37 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:41728
I20260810 23:29:37.124541 9073 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c96bf37396a4a689a695bf4382fe0a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:29:37 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:52572
I20260810 23:29:37.124667 8639 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c96bf37396a4a689a695bf4382fe0a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:29:37 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:43632
I20260810 23:29:37.130074 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 8487
I20260810 23:29:37.144902 9241 tablet_replica.cc:330] T 4c96bf37396a4a689a695bf4382fe0a3 P 2f41eb4f81424f35b0f19a66421a8cd0: stopping tablet replica
I20260810 23:29:37.145534 9241 raft_consensus.cc:2229] T 4c96bf37396a4a689a695bf4382fe0a3 P 2f41eb4f81424f35b0f19a66421a8cd0 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:29:37.146298 9241 raft_consensus.cc:2258] T 4c96bf37396a4a689a695bf4382fe0a3 P 2f41eb4f81424f35b0f19a66421a8cd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:29:37.151288 9203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:37.131849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43646 (local address 127.14.175.65:43593)
0810 23:29:37.132037 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 23:29:37.132043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:29:37.132063 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:29:37.132127 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:29:37.132131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:29:37.132238 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:29:37.132332 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:29:37.134494 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:37.135855 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:37.137957 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:37.138672 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:37.138786 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:37.142580 (+ 3794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:29:37.142608 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:29:37.142635 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:29:37.142691 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:29:37.146954 (+ 4263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:37.147915 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:37.147925 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:37.147929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:37.148114 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:37.148577 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:37.148582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:37.148585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:37.149075 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:29:37.149392 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0810 23:29:37.150625 (+ 1233us) server_negotiation.cc:299] Negotiation successful
0810 23:29:37.150867 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 23:29:37.152150 8737 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:29:37.154021 9241 ts_tablet_manager.cc:1905] T 4c96bf37396a4a689a695bf4382fe0a3 P 2f41eb4f81424f35b0f19a66421a8cd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:29:37.161312 9240 tablet_replica.cc:330] T 4c96bf37396a4a689a695bf4382fe0a3 P dc053cfca0964ebf87c75bd4dee18bc7: stopping tablet replica
I20260810 23:29:37.162004 9241 ts_tablet_manager.cc:1918] T 4c96bf37396a4a689a695bf4382fe0a3 P 2f41eb4f81424f35b0f19a66421a8cd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:29:37.162025 9240 raft_consensus.cc:2229] T 4c96bf37396a4a689a695bf4382fe0a3 P dc053cfca0964ebf87c75bd4dee18bc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:29:37.162177 9241 log.cc:1196] T 4c96bf37396a4a689a695bf4382fe0a3 P 2f41eb4f81424f35b0f19a66421a8cd0: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/wal/wals/4c96bf37396a4a689a695bf4382fe0a3
I20260810 23:29:37.162599 9240 raft_consensus.cc:2258] T 4c96bf37396a4a689a695bf4382fe0a3 P dc053cfca0964ebf87c75bd4dee18bc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:29:37.162684 9241 ts_tablet_manager.cc:1939] T 4c96bf37396a4a689a695bf4382fe0a3 P 2f41eb4f81424f35b0f19a66421a8cd0: Deleting consensus metadata
I20260810 23:29:37.164495 8406 catalog_manager.cc:4841] TS 2f41eb4f81424f35b0f19a66421a8cd0 (127.14.175.67:36617): tablet 4c96bf37396a4a689a695bf4382fe0a3 (table test-table [id=ee92fc1edb8f41be85820c618bbdcf54]) successfully deleted
I20260810 23:29:37.164851 9240 ts_tablet_manager.cc:1905] T 4c96bf37396a4a689a695bf4382fe0a3 P dc053cfca0964ebf87c75bd4dee18bc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:29:37.173005 9240 ts_tablet_manager.cc:1918] T 4c96bf37396a4a689a695bf4382fe0a3 P dc053cfca0964ebf87c75bd4dee18bc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:29:37.173327 9240 log.cc:1196] T 4c96bf37396a4a689a695bf4382fe0a3 P dc053cfca0964ebf87c75bd4dee18bc7: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/wal/wals/4c96bf37396a4a689a695bf4382fe0a3
I20260810 23:29:37.174088 9240 ts_tablet_manager.cc:1939] T 4c96bf37396a4a689a695bf4382fe0a3 P dc053cfca0964ebf87c75bd4dee18bc7: Deleting consensus metadata
I20260810 23:29:37.176240 8405 catalog_manager.cc:4841] TS dc053cfca0964ebf87c75bd4dee18bc7 (127.14.175.66:37175): tablet 4c96bf37396a4a689a695bf4382fe0a3 (table test-table [id=ee92fc1edb8f41be85820c618bbdcf54]) successfully deleted
I20260810 23:29:37.265048 9242 tablet_replica.cc:330] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c: stopping tablet replica
I20260810 23:29:37.265410 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 8487
I20260810 23:29:37.265537 9242 raft_consensus.cc:2229] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:29:37.294081 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 8752
W20260810 23:29:37.294077 8407 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.175.65:43593 (error 108)
I20260810 23:29:37.296443 9245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:37.295750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:43593 (local address 127.0.0.1:43648)
0810 23:29:37.296168 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:37.296279 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:43593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":224,"threads_started":1}
W20260810 23:29:37.296695 8407 catalog_manager.cc:4584] TS 4875e7ba636a4c9fa099a90d2abb0e7c (127.14.175.65:43593): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4c96bf37396a4a689a695bf4382fe0a3: Network error: Client connection negotiation failed: client connection to 127.14.175.65:43593: connect: Connection refused (error 111)
I20260810 23:29:37.310443 9199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:37.295415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41732 (local address 127.14.175.66:37175)
0810 23:29:37.295611 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 23:29:37.295618 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:29:37.295639 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:29:37.295706 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:29:37.295711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:29:37.295791 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:29:37.295894 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:29:37.297936 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:37.298705 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:37.300641 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:37.301034 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:37.301131 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:37.304214 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:29:37.304231 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:29:37.304250 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:29:37.304293 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:29:37.307202 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:37.308041 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:37.308050 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:37.308052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:37.308201 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:37.308587 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:37.308593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:37.308595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:37.309028 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:29:37.309369 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0810 23:29:37.309880 (+ 511us) server_negotiation.cc:299] Negotiation successful
0810 23:29:37.310112 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:29:37.311282 8958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:29:37.350471 9245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:37.349882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:43593 (local address 127.0.0.1:43664)
0810 23:29:37.350161 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:37.350282 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:43593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260810 23:29:37.351835 9245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:37.351385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:43593 (local address 127.0.0.1:43666)
0810 23:29:37.351558 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:37.351654 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:43593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260810 23:29:37.402740 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 8752
I20260810 23:29:37.412664 9245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:37.412158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:43593 (local address 127.0.0.1:43668)
0810 23:29:37.412372 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:37.412472 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:43593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260810 23:29:37.413815 9245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:37.413439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:43593 (local address 127.0.0.1:43678)
0810 23:29:37.413598 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:37.413672 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:43593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 23:29:37.434692 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 8968
I20260810 23:29:37.449564 9204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:37.435617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52578 (local address 127.14.175.67:36617)
0810 23:29:37.435830 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 23:29:37.435840 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:29:37.435859 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:29:37.435928 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:29:37.435932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:29:37.435993 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:29:37.436078 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:29:37.437548 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:37.438486 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:37.439841 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:37.440296 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:37.440414 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:37.443306 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:29:37.443322 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:29:37.443339 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:29:37.443374 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:29:37.446503 (+ 3129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:37.447256 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:37.447264 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:37.447266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:37.447406 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:37.447803 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:37.447807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:37.447809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:37.448217 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:29:37.448511 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0810 23:29:37.448812 (+ 301us) server_negotiation.cc:299] Negotiation successful
0810 23:29:37.449028 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 23:29:37.450350 9177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:29:37.494884 9245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:37.494313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:43593 (local address 127.0.0.1:43680)
0810 23:29:37.494590 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:37.494697 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:43593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260810 23:29:37.496183 9245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:37.495705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:43593 (local address 127.0.0.1:43686)
0810 23:29:37.495923 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:37.496005 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:43593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260810 23:29:37.576323 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 8968
I20260810 23:29:37.602375 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 8380
I20260810 23:29:37.603691 8478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:29:37.720890 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 8380
I20260810 23:29:57.753566 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:45843
--webserver_interface=127.14.175.126
--webserver_port=34573
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:42347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:45843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:29:58.103179 9250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:29:58.103492 9250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:29:58.103641 9250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:29:58.111341 9250 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 23:29:58.111464 9250 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 23:29:58.111511 9250 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 23:29:58.111590 9250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:29:58.111666 9250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:29:58.111773 9250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:29:58.111826 9250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:29:58.111903 9250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:29:58.122277 9250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:42347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:45843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:45843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=34573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:29:58.123746 9250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:29:58.125304 9250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:29:58.137784 9262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:29:58.137823 9265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:29:58.137842 9260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:29:58.139315 9250 server_base.cc:1034] running on GCE node
I20260810 23:29:58.140215 9250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:29:58.141836 9250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:29:58.143097 9250 hybrid_clock.cc:648] HybridClock initialized: now 1786404598143024 us; error 40 us; skew 500 ppm
I20260810 23:29:58.143429 9250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:29:58 dist-test-slave-rmhg krb5kdc[7469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404598, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:29:58.146800 9250 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:29:58.148392 9250 webserver.cc:460] Webserver started at http://127.14.175.126:34573/ using document root <none> and password file <none>
I20260810 23:29:58.149138 9250 fs_manager.cc:359] Metadata directory not provided
I20260810 23:29:58.149300 9250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:29:58.155112 9250 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 23:29:58.158202 9275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:29:58.159624 9250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:29:58.159803 9250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "3f3e6979d111499fb5b7805eabf0241e"
format_stamp: "Formatted at 2026-08-10 23:29:11 on dist-test-slave-rmhg"
server_key: "a1129d012a7ae976ff06d2ec6dde2c51"
server_key_iv: "6d6bf709d15e134d04e2b1560dcab157"
server_key_version: "encryptionkey@0"
I20260810 23:29:58.160828 9250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:29:58.173782 9250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:29:58.178318 9250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:29:58.178606 9250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:29:58.198318 9250 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:45843
I20260810 23:29:58.198324 9358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:45843 every 8 connection(s)
I20260810 23:29:58.200034 9250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:29:58.203996 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 9250
I20260810 23:29:58.204927 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.65:43593
--local_ip_for_outbound_sockets=127.14.175.65
--tserver_master_addrs=127.14.175.126:45843
--webserver_port=46523
--webserver_interface=127.14.175.65
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:42347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:29:58.207943 9359 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:29:58.211639 9359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e: Bootstrap starting.
I20260810 23:29:58.233731 9359 log.cc:824] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:29:58.250721 9359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:29:58.251264 9359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e: Bootstrap complete.
I20260810 23:29:58.260342 9359 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3e6979d111499fb5b7805eabf0241e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 45843 } }
I20260810 23:29:58.261294 9359 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f3e6979d111499fb5b7805eabf0241e, State: Initialized, Role: FOLLOWER
I20260810 23:29:58.261972 9359 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3e6979d111499fb5b7805eabf0241e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 45843 } }
I20260810 23:29:58.262192 9359 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:29:58.262324 9359 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:29:58.262486 9359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [term 2 FOLLOWER]: Advancing to term 3
I20260810 23:29:58.266666 9359 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3e6979d111499fb5b7805eabf0241e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 45843 } }
I20260810 23:29:58.267158 9359 leader_election.cc:302] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f3e6979d111499fb5b7805eabf0241e; no voters:
I20260810 23:29:58.267832 9359 leader_election.cc:288] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 23:29:58.267997 9364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [term 3 FOLLOWER]: Leader election won for term 3
I20260810 23:29:58.270195 9359 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:29:58.274945 9364 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [term 3 LEADER]: Becoming Leader. State: Replica: 3f3e6979d111499fb5b7805eabf0241e, State: Running, Role: LEADER
I20260810 23:29:58.275624 9364 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3e6979d111499fb5b7805eabf0241e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 45843 } }
I20260810 23:29:58.282600 9367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f3e6979d111499fb5b7805eabf0241e. Latest consensus state: current_term: 3 leader_uuid: "3f3e6979d111499fb5b7805eabf0241e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3e6979d111499fb5b7805eabf0241e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 45843 } } }
I20260810 23:29:58.283044 9367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [sys.catalog]: This master's current role is: LEADER
I20260810 23:29:58.283640 9366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3f3e6979d111499fb5b7805eabf0241e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3e6979d111499fb5b7805eabf0241e" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 45843 } } }
I20260810 23:29:58.283941 9366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e [sys.catalog]: This master's current role is: LEADER
I20260810 23:29:58.290459 9383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:29:58.299609 9383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:29:58.301219 9383 catalog_manager.cc:1259] Loaded cluster ID: 506ab82d48304dcd94becd57e18afe2a
I20260810 23:29:58.301376 9383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:29:58.305552 9383 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:29:58.306422 9389 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 23:29:58.307479 9383 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e: Loaded TSK: 0
I20260810 23:29:58.307638 9383 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3f3e6979d111499fb5b7805eabf0241e: Loaded TSK: 1
F20260810 23:29:58.308844 9383 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f564ae619ad google::LogMessage::Fail() at ??:0
@ 0x7f564ae6488a google::LogMessage::SendToLog() at ??:0
@ 0x7f564ae614de google::LogMessage::Flush() at ??:0
@ 0x7f564ae62a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f564ebc57d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f564ebb473e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f56608941c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f5660893783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f566091f2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f566091f0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f56609695e9 std::function<>::operator()() at ??:0
@ 0x7f5660897cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f5660895c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f566091d610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f566091d439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f564daad345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f564dab091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f564dab0799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f564da8beaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5650d656db start_thread at ??:0
@ 0x7f5649cfb61f clone at ??:0
W20260810 23:29:58.618553 9362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:29:58.618880 9362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:29:58.619006 9362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:29:58.626480 9362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:29:58.626636 9362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:29:58.626788 9362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.65
I20260810 23:29:58.637354 9362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:42347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.65:43593
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.175.65
--webserver_port=46523
--tserver_master_addrs=127.14.175.126:45843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.65
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:29:58.638713 9362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:29:58.640044 9362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:29:58.653537 9428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:29:58.653508 9431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:29:58.653601 9433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:29:58.654240 9362 server_base.cc:1034] running on GCE node
I20260810 23:29:58.654954 9362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:29:58.656023 9362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:29:58.657260 9362 hybrid_clock.cc:648] HybridClock initialized: now 1786404598657194 us; error 48 us; skew 500 ppm
I20260810 23:29:58.657608 9362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:29:58 dist-test-slave-rmhg krb5kdc[7469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404598, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:29:58.660720 9362 init.cc:377] Logged in from keytab as kudu/127.14.175.65@KRBTEST.COM (short username kudu)
I20260810 23:29:58.662292 9362 webserver.cc:460] Webserver started at http://127.14.175.65:46523/ using document root <none> and password file <none>
I20260810 23:29:58.662986 9362 fs_manager.cc:359] Metadata directory not provided
I20260810 23:29:58.663101 9362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:29:58.669046 9362 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 23:29:58.672348 9451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:29:58.673888 9362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 23:29:58.674052 9362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/wal
uuid: "4875e7ba636a4c9fa099a90d2abb0e7c"
format_stamp: "Formatted at 2026-08-10 23:29:12 on dist-test-slave-rmhg"
server_key: "cc6fb69dac1033ba3e7baba662a4ba6e"
server_key_iv: "03536a959811b9e3f943b60185fbc337"
server_key_version: "encryptionkey@0"
I20260810 23:29:58.675076 9362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:29:58.693375 9362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:29:58.697755 9362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:29:58.698031 9362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:29:58.699091 9362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:29:58.701040 9462 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260810 23:29:58.707368 9362 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 23:29:58.707507 9362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260810 23:29:58.707635 9362 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 23:29:58.710922 9362 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 23:29:58.711030 9362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260810 23:29:58.711381 9462 tablet_bootstrap.cc:492] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c: Bootstrap starting.
I20260810 23:29:58.739553 9462 log.cc:824] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c: Log is configured to *not* fsync() on all Append() calls
I20260810 23:29:58.751435 9362 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.65:43593
I20260810 23:29:58.751472 9637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.65:43593 every 8 connection(s)
I20260810 23:29:58.753211 9362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
I20260810 23:29:58.753778 9462 tablet_bootstrap.cc:492] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:29:58.754585 9462 tablet_bootstrap.cc:492] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c: Bootstrap complete.
I20260810 23:29:58.756174 9462 ts_tablet_manager.cc:1397] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c: Time spent bootstrapping tablet: real 0.045s user 0.021s sys 0.021s
I20260810 23:29:58.757342 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 9362
I20260810 23:29:58.758219 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.66:37175
--local_ip_for_outbound_sockets=127.14.175.66
--tserver_master_addrs=127.14.175.126:45843
--webserver_port=36815
--webserver_interface=127.14.175.66
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:42347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:29:58.759354 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:58.756390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.65:44583)
0810 23:29:58.757576 (+ 1186us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:58.757934 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":475,"threads_started":1}
W20260810 23:29:58.760309 9638 heartbeater.cc:644] Failed to heartbeat to 127.14.175.126:45843 (0 consecutive failures): Network error: Failed to ping master at 127.14.175.126:45843: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
I20260810 23:29:58.761447 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:58.760972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.65:39621)
0810 23:29:58.761186 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:58.761288 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 23:29:58.762677 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:58.762218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.65:44285)
0810 23:29:58.762382 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:58.762498 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260810 23:29:58.762962 9638 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 23:29:58.764663 9462 raft_consensus.cc:348] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc053cfca0964ebf87c75bd4dee18bc7" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 37175 } } peers { permanent_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 36617 } } peers { permanent_uuid: "4875e7ba636a4c9fa099a90d2abb0e7c" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 43593 } }
I20260810 23:29:58.765679 9462 raft_consensus.cc:729] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4875e7ba636a4c9fa099a90d2abb0e7c, State: Initialized, Role: FOLLOWER
I20260810 23:29:58.766317 9462 consensus_queue.cc:260] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc053cfca0964ebf87c75bd4dee18bc7" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 37175 } } peers { permanent_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 36617 } } peers { permanent_uuid: "4875e7ba636a4c9fa099a90d2abb0e7c" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 43593 } }
I20260810 23:29:58.767961 9462 ts_tablet_manager.cc:1428] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260810 23:29:58.768196 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:58.767734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.65:49577)
0810 23:29:58.767925 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:58.768033 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 23:29:58.769234 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:58.768854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.65:35083)
0810 23:29:58.768986 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:58.769062 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 23:29:59.119436 9648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:29:59.119787 9648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:29:59.119927 9648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:29:59.127700 9648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:29:59.127883 9648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:29:59.128054 9648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.66
I20260810 23:29:59.138679 9648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:42347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.66:37175
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.175.66
--webserver_port=36815
--tserver_master_addrs=127.14.175.126:45843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.66
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:29:59.140158 9648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:29:59.141654 9648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:29:59.155988 9658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:29:59.156014 9660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:29:59.155989 9654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:29:59.156716 9648 server_base.cc:1034] running on GCE node
I20260810 23:29:59.157516 9648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:29:59.158711 9648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:29:59.159941 9648 hybrid_clock.cc:648] HybridClock initialized: now 1786404599159865 us; error 72 us; skew 500 ppm
I20260810 23:29:59.160305 9648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:29:59 dist-test-slave-rmhg krb5kdc[7469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404599, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:29:59.163772 9648 init.cc:377] Logged in from keytab as kudu/127.14.175.66@KRBTEST.COM (short username kudu)
I20260810 23:29:59.165674 9648 webserver.cc:460] Webserver started at http://127.14.175.66:36815/ using document root <none> and password file <none>
I20260810 23:29:59.166440 9648 fs_manager.cc:359] Metadata directory not provided
I20260810 23:29:59.166563 9648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:29:59.172816 9648 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:29:59.175938 9673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:29:59.177584 9648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 23:29:59.177774 9648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/wal
uuid: "dc053cfca0964ebf87c75bd4dee18bc7"
format_stamp: "Formatted at 2026-08-10 23:29:12 on dist-test-slave-rmhg"
server_key: "d2aa37f0ba78c8eda5f8708686eadfa9"
server_key_iv: "099366519a0caba015482782ac9b470a"
server_key_version: "encryptionkey@0"
I20260810 23:29:59.178833 9648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:29:59.193430 9648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:29:59.197983 9648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:29:59.198277 9648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:29:59.199365 9648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:29:59.200819 9648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:29:59.200917 9648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:29:59.201030 9648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:29:59.201104 9648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:29:59.233805 9648 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.66:37175
I20260810 23:29:59.233880 9847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.66:37175 every 8 connection(s)
I20260810 23:29:59.235349 9648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
I20260810 23:29:59.240396 9852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:59.238451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.66:38435)
0810 23:29:59.239229 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:59.239483 (+ 254us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":291,"threads_started":1}
W20260810 23:29:59.241237 9848 heartbeater.cc:644] Failed to heartbeat to 127.14.175.126:45843 (0 consecutive failures): Network error: Failed to ping master at 127.14.175.126:45843: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
I20260810 23:29:59.241719 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 9648
I20260810 23:29:59.242357 9852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:59.241913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.66:58733)
0810 23:29:59.242105 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:59.242204 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260810 23:29:59.242480 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.67:36617
--local_ip_for_outbound_sockets=127.14.175.67
--tserver_master_addrs=127.14.175.126:45843
--webserver_port=37915
--webserver_interface=127.14.175.67
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:42347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:29:59.243274 9852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:59.242939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.66:43099)
0810 23:29:59.243072 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:59.243158 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 23:29:59.243525 9848 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 23:29:59.598336 9854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:29:59.598660 9854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:29:59.598784 9854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:29:59.606459 9854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:29:59.606598 9854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:29:59.606756 9854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.67
I20260810 23:29:59.617081 9854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:42347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.67:36617
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.175.67
--webserver_port=37915
--tserver_master_addrs=127.14.175.126:45843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.67
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:29:59.618494 9854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:29:59.619874 9854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:29:59.633540 9866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:29:59.633594 9867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:29:59.633525 9869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:29:59.634781 9854 server_base.cc:1034] running on GCE node
I20260810 23:29:59.635510 9854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:29:59.636560 9854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:29:59.637820 9854 hybrid_clock.cc:648] HybridClock initialized: now 1786404599637758 us; error 47 us; skew 500 ppm
I20260810 23:29:59.638198 9854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:29:59 dist-test-slave-rmhg krb5kdc[7469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404599, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:29:59.644696 9854 init.cc:377] Logged in from keytab as kudu/127.14.175.67@KRBTEST.COM (short username kudu)
I20260810 23:29:59.646450 9854 webserver.cc:460] Webserver started at http://127.14.175.67:37915/ using document root <none> and password file <none>
I20260810 23:29:59.647171 9854 fs_manager.cc:359] Metadata directory not provided
I20260810 23:29:59.647285 9854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:29:59.653221 9854 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260810 23:29:59.656209 9879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:29:59.657721 9854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 23:29:59.657900 9854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/wal
uuid: "2f41eb4f81424f35b0f19a66421a8cd0"
format_stamp: "Formatted at 2026-08-10 23:29:13 on dist-test-slave-rmhg"
server_key: "dde0da0c43fcf4d8c1a7178d0c4bd29f"
server_key_iv: "9d9e62a9d4195e91536755027ee856ff"
server_key_version: "encryptionkey@0"
I20260810 23:29:59.658933 9854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:29:59.697456 9854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:29:59.702325 9854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:29:59.702636 9854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:29:59.703815 9854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:29:59.705417 9854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:29:59.705526 9854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:29:59.705638 9854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:29:59.705704 9854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:29:59.739108 9854 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.67:36617
I20260810 23:29:59.739171 10061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.67:36617 every 8 connection(s)
I20260810 23:29:59.740703 9854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
I20260810 23:29:59.744115 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 9854
I20260810 23:29:59.746286 10066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:59.744050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.67:39661)
0810 23:29:59.744924 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:59.745266 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":353,"threads_started":1}
W20260810 23:29:59.747051 10062 heartbeater.cc:644] Failed to heartbeat to 127.14.175.126:45843 (0 consecutive failures): Network error: Failed to ping master at 127.14.175.126:45843: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
I20260810 23:29:59.747932 10066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:59.747589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.67:49083)
0810 23:29:59.747747 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:59.747814 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 23:29:59.748732 10066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:59.748410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.67:33723)
0810 23:29:59.748553 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:59.748626 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 23:29:59.748970 10062 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 23:29:59.770918 10069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:59.770081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.65:39951)
0810 23:29:59.770590 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:59.770705 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":244,"threads_started":1}
I20260810 23:29:59.961133 10070 raft_consensus.cc:482] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:29:59.961496 10070 raft_consensus.cc:504] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc053cfca0964ebf87c75bd4dee18bc7" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 37175 } } peers { permanent_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 36617 } } peers { permanent_uuid: "4875e7ba636a4c9fa099a90d2abb0e7c" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 43593 } }
I20260810 23:29:59.963218 10070 leader_election.cc:288] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dc053cfca0964ebf87c75bd4dee18bc7 (127.14.175.66:37175), 2f41eb4f81424f35b0f19a66421a8cd0 (127.14.175.67:36617)
Aug 10 23:29:59 dist-test-slave-rmhg krb5kdc[7469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404598, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for kudu/127.14.175.66@KRBTEST.COM
Aug 10 23:29:59 dist-test-slave-rmhg krb5kdc[7469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404598, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for kudu/127.14.175.67@KRBTEST.COM
I20260810 23:29:59.985342 10072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:59.964039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:36617 (local address 127.14.175.65:43843)
0810 23:29:59.965563 (+ 1524us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:59.965581 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:29:59.965707 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:29:59.967974 (+ 2267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:29:59.967979 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:29:59.968606 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:29:59.969300 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:59.969322 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:59.970932 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:59.970936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:59.972637 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:59.972652 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:59.974516 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:59.974521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:59.974648 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:59.975247 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:29:59.975265 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:29:59.978094 (+ 2829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:29:59.981604 (+ 3510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:29:59.981610 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:29:59.981614 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:29:59.982232 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:29:59.982725 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:29:59.982730 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:29:59.982733 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:29:59.982917 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:29:59.983887 (+ 970us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:29:59.983895 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:29:59.984216 (+ 321us) client_negotiation.cc:769] Sending connection context
0810 23:29:59.984485 (+ 269us) client_negotiation.cc:240] Negotiation successful
0810 23:29:59.984924 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1399,"mutex_wait_us":117,"thread_start_us":1202,"threads_started":1}
I20260810 23:29:59.986094 10069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:59.963686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:37175 (local address 127.14.175.65:57541)
0810 23:29:59.963884 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 23:29:59.963952 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 23:29:59.964591 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:29:59.967676 (+ 3085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:29:59.967689 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:29:59.968486 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:29:59.969299 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:59.969322 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:59.970930 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:59.970935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:59.972666 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:29:59.972683 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:29:59.974207 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:29:59.974212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:29:59.974419 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:59.975156 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:29:59.975199 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:29:59.977868 (+ 2669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:29:59.981555 (+ 3687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:29:59.981565 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:29:59.981581 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:29:59.982210 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:29:59.982671 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:29:59.982703 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:29:59.982705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:29:59.982923 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:29:59.984666 (+ 1743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:29:59.984671 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:29:59.984837 (+ 166us) client_negotiation.cc:769] Sending connection context
0810 23:29:59.984998 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 23:29:59.985304 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":189}
I20260810 23:29:59.986268 10071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:59.963761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:57541 (local address 127.14.175.66:37175)
0810 23:29:59.964378 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0810 23:29:59.964397 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 23:29:59.964453 (+ 56us) server_negotiation.cc:407] Connection header received
0810 23:29:59.966786 (+ 2333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:29:59.966797 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:29:59.967204 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:29:59.967548 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:29:59.969503 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:59.970732 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:59.972857 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:59.973977 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:59.974183 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:59.978279 (+ 4096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:29:59.978289 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:29:59.978309 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:29:59.978358 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:29:59.981400 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:59.982346 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:59.982354 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:59.982360 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:59.982548 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:59.983029 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:59.983032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:59.983035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:59.984530 (+ 1495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:29:59.984699 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 23:29:59.985299 (+ 600us) server_negotiation.cc:299] Negotiation successful
0810 23:29:59.985854 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":759,"threads_started":1}
I20260810 23:29:59.986586 10073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:29:59.964277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:43843 (local address 127.14.175.67:36617)
0810 23:29:59.965276 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0810 23:29:59.965290 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:29:59.965683 (+ 393us) server_negotiation.cc:407] Connection header received
0810 23:29:59.967340 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:29:59.967348 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:29:59.967699 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:29:59.967974 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:29:59.969502 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:59.970722 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:59.972940 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:29:59.973989 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:29:59.974195 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:29:59.978383 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:29:59.978392 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:29:59.978405 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:29:59.978450 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:29:59.981460 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:59.982420 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:59.982427 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:59.982433 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:59.982603 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:29:59.983043 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:29:59.983049 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:29:59.983053 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:29:59.983732 (+ 679us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:29:59.983912 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 23:29:59.985847 (+ 1935us) server_negotiation.cc:299] Negotiation successful
0810 23:29:59.986185 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":773,"thread_start_us":594,"threads_started":1}
W20260810 23:29:59.989871 9453 leader_election.cc:341] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2f41eb4f81424f35b0f19a66421a8cd0 (127.14.175.67:36617): Not found: Tablet not found: 4c96bf37396a4a689a695bf4382fe0a3
W20260810 23:29:59.990175 9452 leader_election.cc:341] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dc053cfca0964ebf87c75bd4dee18bc7 (127.14.175.66:37175): Not found: Tablet not found: 4c96bf37396a4a689a695bf4382fe0a3
I20260810 23:29:59.990310 9452 leader_election.cc:302] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4875e7ba636a4c9fa099a90d2abb0e7c; no voters: 2f41eb4f81424f35b0f19a66421a8cd0, dc053cfca0964ebf87c75bd4dee18bc7
I20260810 23:29:59.990697 10070 raft_consensus.cc:2726] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:30:00.245319 10078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:00.244256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.66:43033)
0810 23:30:00.244847 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:00.245014 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":254,"threads_started":1}
I20260810 23:30:00.750561 10080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:00.749714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.67:43765)
0810 23:30:00.750251 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:00.750359 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":243,"threads_started":1}
I20260810 23:30:00.773011 10081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:00.771996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.65:44383)
0810 23:30:00.772675 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:00.772783 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":323,"threads_started":1}
I20260810 23:30:01.247087 10082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:01.246332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.66:47861)
0810 23:30:01.246822 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:01.246920 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":248,"threads_started":1}
I20260810 23:30:01.664414 10083 raft_consensus.cc:482] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:30:01.664676 10083 raft_consensus.cc:504] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc053cfca0964ebf87c75bd4dee18bc7" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 37175 } } peers { permanent_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 36617 } } peers { permanent_uuid: "4875e7ba636a4c9fa099a90d2abb0e7c" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 43593 } }
I20260810 23:30:01.665424 10083 leader_election.cc:288] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dc053cfca0964ebf87c75bd4dee18bc7 (127.14.175.66:37175), 2f41eb4f81424f35b0f19a66421a8cd0 (127.14.175.67:36617)
W20260810 23:30:01.666447 9453 leader_election.cc:341] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2f41eb4f81424f35b0f19a66421a8cd0 (127.14.175.67:36617): Not found: Tablet not found: 4c96bf37396a4a689a695bf4382fe0a3
W20260810 23:30:01.666625 9452 leader_election.cc:341] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dc053cfca0964ebf87c75bd4dee18bc7 (127.14.175.66:37175): Not found: Tablet not found: 4c96bf37396a4a689a695bf4382fe0a3
I20260810 23:30:01.666761 9452 leader_election.cc:302] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4875e7ba636a4c9fa099a90d2abb0e7c; no voters: 2f41eb4f81424f35b0f19a66421a8cd0, dc053cfca0964ebf87c75bd4dee18bc7
I20260810 23:30:01.667026 10083 raft_consensus.cc:2726] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:30:01.752539 10084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:01.751623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.67:42831)
0810 23:30:01.752202 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:01.752318 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":284,"threads_started":1}
I20260810 23:30:01.775249 10088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:01.774114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.65:40207)
0810 23:30:01.774790 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:01.774944 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":276,"threads_started":1}
I20260810 23:30:02.248999 10089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:02.248154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.66:33343)
0810 23:30:02.248704 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:02.248805 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":274,"threads_started":1}
I20260810 23:30:02.754308 10090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:02.753577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.67:49921)
0810 23:30:02.754040 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:02.754140 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":241,"threads_started":1}
I20260810 23:30:02.777102 10091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:02.776339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.65:40331)
0810 23:30:02.776840 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:02.776929 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":244,"threads_started":1}
I20260810 23:30:03.250964 10093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:03.250091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.66:57577)
0810 23:30:03.250649 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:03.250770 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":256,"threads_started":1}
I20260810 23:30:03.756060 10096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:03.755304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.67:45457)
0810 23:30:03.755789 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:03.755888 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":227,"threads_started":1}
I20260810 23:30:03.779261 10097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:03.778420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.65:36145)
0810 23:30:03.778948 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:03.779062 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":271,"threads_started":1}
I20260810 23:30:04.252743 10099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:04.251971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.66:55835)
0810 23:30:04.252467 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:04.252557 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":268,"threads_started":1}
I20260810 23:30:04.434604 10103 raft_consensus.cc:482] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:30:04.434811 10103 raft_consensus.cc:504] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc053cfca0964ebf87c75bd4dee18bc7" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 37175 } } peers { permanent_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 36617 } } peers { permanent_uuid: "4875e7ba636a4c9fa099a90d2abb0e7c" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 43593 } }
I20260810 23:30:04.435511 10103 leader_election.cc:288] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dc053cfca0964ebf87c75bd4dee18bc7 (127.14.175.66:37175), 2f41eb4f81424f35b0f19a66421a8cd0 (127.14.175.67:36617)
W20260810 23:30:04.436617 9453 leader_election.cc:341] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2f41eb4f81424f35b0f19a66421a8cd0 (127.14.175.67:36617): Not found: Tablet not found: 4c96bf37396a4a689a695bf4382fe0a3
W20260810 23:30:04.436823 9452 leader_election.cc:341] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dc053cfca0964ebf87c75bd4dee18bc7 (127.14.175.66:37175): Not found: Tablet not found: 4c96bf37396a4a689a695bf4382fe0a3
I20260810 23:30:04.436926 9452 leader_election.cc:302] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4875e7ba636a4c9fa099a90d2abb0e7c; no voters: 2f41eb4f81424f35b0f19a66421a8cd0, dc053cfca0964ebf87c75bd4dee18bc7
I20260810 23:30:04.437182 10103 raft_consensus.cc:2726] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:30:04.757987 10104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:04.757129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.67:35789)
0810 23:30:04.757732 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:04.757823 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":294,"threads_started":1}
I20260810 23:30:04.781131 10106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:04.780265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.65:49817)
0810 23:30:04.780819 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:04.780929 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":272,"threads_started":1}
I20260810 23:30:05.254602 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:05.253792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.66:55799)
0810 23:30:05.254309 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:05.254404 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":268,"threads_started":1}
I20260810 23:30:05.759708 10108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:05.758928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.67:41581)
0810 23:30:05.759427 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:05.759519 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":248,"threads_started":1}
I20260810 23:30:05.783077 10112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:05.782114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.65:54175)
0810 23:30:05.782693 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:05.782820 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":260,"threads_started":1}
I20260810 23:30:06.256626 10113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:06.255713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.66:52559)
0810 23:30:06.256314 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:06.256416 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":259,"threads_started":1}
I20260810 23:30:06.580617 10115 raft_consensus.cc:482] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:30:06.580855 10115 raft_consensus.cc:504] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc053cfca0964ebf87c75bd4dee18bc7" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 37175 } } peers { permanent_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 36617 } } peers { permanent_uuid: "4875e7ba636a4c9fa099a90d2abb0e7c" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 43593 } }
I20260810 23:30:06.581529 10115 leader_election.cc:288] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dc053cfca0964ebf87c75bd4dee18bc7 (127.14.175.66:37175), 2f41eb4f81424f35b0f19a66421a8cd0 (127.14.175.67:36617)
W20260810 23:30:06.582501 9453 leader_election.cc:341] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2f41eb4f81424f35b0f19a66421a8cd0 (127.14.175.67:36617): Not found: Tablet not found: 4c96bf37396a4a689a695bf4382fe0a3
W20260810 23:30:06.582729 9452 leader_election.cc:341] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dc053cfca0964ebf87c75bd4dee18bc7 (127.14.175.66:37175): Not found: Tablet not found: 4c96bf37396a4a689a695bf4382fe0a3
I20260810 23:30:06.582842 9452 leader_election.cc:302] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4875e7ba636a4c9fa099a90d2abb0e7c; no voters: 2f41eb4f81424f35b0f19a66421a8cd0, dc053cfca0964ebf87c75bd4dee18bc7
I20260810 23:30:06.583103 10115 raft_consensus.cc:2726] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:30:06.761731 10116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:06.760788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.67:32993)
0810 23:30:06.761396 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:06.761491 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":271,"threads_started":1}
I20260810 23:30:06.784971 10117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:06.784080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.65:43987)
0810 23:30:06.784644 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:06.784768 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":299,"threads_started":1}
I20260810 23:30:07.258523 10121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:07.257763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.66:44647)
0810 23:30:07.258224 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:07.258319 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":226,"threads_started":1}
I20260810 23:30:07.763491 10122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:07.762737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.67:49341)
0810 23:30:07.763208 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:07.763309 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":220,"threads_started":1}
I20260810 23:30:07.786885 10125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:07.786057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.65:54801)
0810 23:30:07.786577 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:07.786689 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":266,"threads_started":1}
I20260810 23:30:08.260437 10127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:08.259557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.66:35445)
0810 23:30:08.260158 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:08.260252 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":255,"threads_started":1}
I20260810 23:30:08.765347 10129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:08.764440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.67:60815)
0810 23:30:08.765014 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:08.765118 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":281,"threads_started":1}
I20260810 23:30:08.788993 10131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:08.788161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.65:52501)
0810 23:30:08.788717 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:08.788814 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":262,"threads_started":1}
I20260810 23:30:09.262282 10132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:09.261476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.66:56913)
0810 23:30:09.261960 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:09.262058 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":251,"threads_started":1}
I20260810 23:30:09.767146 10134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:09.766385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.67:50961)
0810 23:30:09.766863 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:09.766960 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":250,"threads_started":1}
I20260810 23:30:09.791036 10135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:09.790130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.65:48337)
0810 23:30:09.790709 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:09.790813 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":293,"threads_started":1}
I20260810 23:30:10.264017 10136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:10.263211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.66:58897)
0810 23:30:10.263755 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:10.263850 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":250,"threads_started":1}
I20260810 23:30:10.264073 10137 raft_consensus.cc:482] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:30:10.264318 10137 raft_consensus.cc:504] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc053cfca0964ebf87c75bd4dee18bc7" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 37175 } } peers { permanent_uuid: "2f41eb4f81424f35b0f19a66421a8cd0" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 36617 } } peers { permanent_uuid: "4875e7ba636a4c9fa099a90d2abb0e7c" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 43593 } }
I20260810 23:30:10.265040 10137 leader_election.cc:288] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dc053cfca0964ebf87c75bd4dee18bc7 (127.14.175.66:37175), 2f41eb4f81424f35b0f19a66421a8cd0 (127.14.175.67:36617)
W20260810 23:30:10.265959 9452 leader_election.cc:341] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dc053cfca0964ebf87c75bd4dee18bc7 (127.14.175.66:37175): Not found: Tablet not found: 4c96bf37396a4a689a695bf4382fe0a3
W20260810 23:30:10.266139 9453 leader_election.cc:341] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2f41eb4f81424f35b0f19a66421a8cd0 (127.14.175.67:36617): Not found: Tablet not found: 4c96bf37396a4a689a695bf4382fe0a3
I20260810 23:30:10.266263 9453 leader_election.cc:302] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4875e7ba636a4c9fa099a90d2abb0e7c; no voters: 2f41eb4f81424f35b0f19a66421a8cd0, dc053cfca0964ebf87c75bd4dee18bc7
I20260810 23:30:10.266531 10137 raft_consensus.cc:2726] T 4c96bf37396a4a689a695bf4382fe0a3 P 4875e7ba636a4c9fa099a90d2abb0e7c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:30:10.768944 10139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:10.768134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.67:43437)
0810 23:30:10.768644 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:10.768768 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":223,"threads_started":1}
I20260810 23:30:10.793002 10141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:10.792151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.65:46073)
0810 23:30:10.792694 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:10.792811 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":248,"threads_started":1}
I20260810 23:30:11.265913 10142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:11.265077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.66:47645)
0810 23:30:11.265608 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:11.265730 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":289,"threads_started":1}
I20260810 23:30:11.770984 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:11.770105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.67:41901)
0810 23:30:11.770703 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:11.770801 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":316,"threads_started":1}
I20260810 23:30:11.794926 10145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:11.794077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.65:37387)
0810 23:30:11.794612 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:11.794725 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":240,"threads_started":1}
I20260810 23:30:12.267755 10147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:12.266926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.66:54947)
0810 23:30:12.267462 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:12.267577 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":287,"threads_started":1}
I20260810 23:30:12.772749 10148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:12.771970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.67:44285)
0810 23:30:12.772460 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:12.772571 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":248,"threads_started":1}
I20260810 23:30:12.796952 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:12.796066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.65:34493)
0810 23:30:12.796637 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:12.796733 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":279,"threads_started":1}
I20260810 23:30:13.269626 10151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:13.268785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.66:37177)
0810 23:30:13.269329 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:13.269420 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":278,"threads_started":1}
I20260810 23:30:13.774621 10153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:13.773825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.67:36235)
0810 23:30:13.774321 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:13.774417 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":248,"threads_started":1}
I20260810 23:30:13.799213 10154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:13.798341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.65:47563)
0810 23:30:13.798904 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:13.799012 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":263,"threads_started":1}
I20260810 23:30:14.271608 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:14.270860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.66:56673)
0810 23:30:14.271317 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:14.271448 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":223,"threads_started":1}
I20260810 23:30:14.754740 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 9362
I20260810 23:30:14.773114 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:14.756072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59742 (local address 127.14.175.65:43593)
0810 23:30:14.757684 (+ 1612us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:14.757692 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:14.757720 (+ 28us) server_negotiation.cc:407] Connection header received
0810 23:30:14.757816 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:14.757823 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:14.758047 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:14.758233 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:14.759747 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:14.760817 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:14.762250 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:14.762910 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:14.763079 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:14.766112 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:14.766131 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:14.766150 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:30:14.766194 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:14.769878 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:14.770805 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:14.770814 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:14.770816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:14.771001 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:14.771434 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:14.771439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:14.771441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:14.771896 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:14.772259 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:14.772581 (+ 322us) server_negotiation.cc:299] Negotiation successful
0810 23:30:14.772833 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1410,"thread_start_us":1269,"threads_started":1}
I20260810 23:30:14.774565 9629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:30:14.776919 10159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:14.775903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:45843 (local address 127.14.175.67:37197)
0810 23:30:14.776498 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:14.776679 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.126:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":267,"threads_started":1}
I20260810 23:30:14.866247 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 9362
I20260810 23:30:14.897830 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 9648
I20260810 23:30:14.912153 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:14.898766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34338 (local address 127.14.175.66:37175)
0810 23:30:14.899261 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:14.899266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:14.899284 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:30:14.899341 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:14.899345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:14.899405 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:14.899495 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:14.900896 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:14.901757 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:14.903135 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:14.903642 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:14.903758 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:14.906655 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:14.906661 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:14.906665 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:30:14.906705 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:14.909391 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:14.910163 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:14.910169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:14.910171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:14.910331 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:14.910671 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:14.910675 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:14.910677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:14.911007 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:14.911238 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:14.911633 (+ 395us) server_negotiation.cc:299] Negotiation successful
0810 23:30:14.911842 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":251,"threads_started":1}
I20260810 23:30:14.912931 9843 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:30:14.997785 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 9648
I20260810 23:30:15.025276 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 9854
I20260810 23:30:15.039778 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:15.026241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43954 (local address 127.14.175.67:36617)
0810 23:30:15.026825 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:15.026830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:15.026847 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:30:15.026903 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:15.026907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:15.026969 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:15.027071 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:15.028608 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:15.029612 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:15.030972 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:15.031423 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:15.031546 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:15.034326 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:15.034332 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:15.034335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:30:15.034373 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:15.036914 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:15.037741 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:15.037746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:15.037749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:15.037902 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:15.038289 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:15.038293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:15.038295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:15.038571 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:15.038862 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:15.039150 (+ 288us) server_negotiation.cc:299] Negotiation successful
0810 23:30:15.039394 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":295,"threads_started":1}
I20260810 23:30:15.040675 10054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:30:15.127415 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 9854
2026-08-10T23:30:15Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65452 ms)
[----------] 30 tests from SecurityITest (315557 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:30:15 dist-test-slave-rmhg krb5kdc[10170](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:30:15 dist-test-slave-rmhg krb5kdc[10170](info): set up 2 sockets
Aug 10 23:30:15 dist-test-slave-rmhg krb5kdc[10170](info): commencing operation
krb5kdc: starting...
W20260810 23:30:16.372130 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.192s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:30:16 dist-test-slave-rmhg krb5kdc[10170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:30:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:30:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
I20260810 23:30:16.688604 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:37827
--webserver_interface=127.14.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:46307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:37827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:30:17.049448 10191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:30:17.049768 10191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:30:17.049906 10191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:30:17.057446 10191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:30:17.057582 10191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:30:17.057664 10191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:30:17.057742 10191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:30:17.057808 10191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:30:17.068431 10191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:46307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:37827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:37827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:30:17.069974 10191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:30:17.071388 10191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:30:17.084107 10197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:17.084153 10198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:17.084193 10200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:30:17.084857 10191 server_base.cc:1034] running on GCE node
I20260810 23:30:17.085729 10191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:30:17.087255 10191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:30:17.088474 10191 hybrid_clock.cc:648] HybridClock initialized: now 1786404617088430 us; error 34 us; skew 500 ppm
I20260810 23:30:17.088850 10191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:30:17 dist-test-slave-rmhg krb5kdc[10170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404617, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:30:17.092090 10191 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:30:17.093889 10191 webserver.cc:460] Webserver started at http://127.14.175.126:42095/ using document root <none> and password file <none>
I20260810 23:30:17.094656 10191 fs_manager.cc:359] Metadata directory not provided
I20260810 23:30:17.094790 10191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:30:17.095112 10191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:30:17.098445 10191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "195a406933e84fd9b05a406ee8848425"
format_stamp: "Formatted at 2026-08-10 23:30:17 on dist-test-slave-rmhg"
server_key: "1ec9f57df59656c110248e3ce055bba5"
server_key_iv: "6b66d3aa7d033fce0f330217faa1d2c4"
server_key_version: "encryptionkey@0"
I20260810 23:30:17.099272 10191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "195a406933e84fd9b05a406ee8848425"
format_stamp: "Formatted at 2026-08-10 23:30:17 on dist-test-slave-rmhg"
server_key: "1ec9f57df59656c110248e3ce055bba5"
server_key_iv: "6b66d3aa7d033fce0f330217faa1d2c4"
server_key_version: "encryptionkey@0"
I20260810 23:30:17.105070 10191 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 23:30:17.109735 10209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:17.111446 10191 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260810 23:30:17.111639 10191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "195a406933e84fd9b05a406ee8848425"
format_stamp: "Formatted at 2026-08-10 23:30:17 on dist-test-slave-rmhg"
server_key: "1ec9f57df59656c110248e3ce055bba5"
server_key_iv: "6b66d3aa7d033fce0f330217faa1d2c4"
server_key_version: "encryptionkey@0"
I20260810 23:30:17.111848 10191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:30:17.131843 10191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:30:17.136664 10191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:30:17.136983 10191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:30:17.156397 10191 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:37827
I20260810 23:30:17.156405 10288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:37827 every 8 connection(s)
I20260810 23:30:17.158015 10191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:30:17.161357 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 10191
I20260810 23:30:17.161602 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:30:17.162274 15037 external_mini_cluster.cc:1428] Setting key 34e3df57dfbc7ceb3a0ea416ca7f918f
I20260810 23:30:17.162389 10289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:30:17.177589 10289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 195a406933e84fd9b05a406ee8848425: Bootstrap starting.
Aug 10 23:30:17 dist-test-slave-rmhg krb5kdc[10170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:30:17.181602 10289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 195a406933e84fd9b05a406ee8848425: Neither blocks nor log segments found. Creating new log.
I20260810 23:30:17.182840 10289 log.cc:824] T 00000000000000000000000000000000 P 195a406933e84fd9b05a406ee8848425: Log is configured to *not* fsync() on all Append() calls
I20260810 23:30:17.186098 10289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 195a406933e84fd9b05a406ee8848425: No bootstrap required, opened a new log
I20260810 23:30:17.187680 10292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:17.164482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60096 (local address 127.14.175.126:37827)
0810 23:30:17.165473 (+ 991us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:17.165492 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:17.165554 (+ 62us) server_negotiation.cc:407] Connection header received
0810 23:30:17.167761 (+ 2207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:17.167798 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:17.168454 (+ 656us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:17.169007 (+ 553us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:17.170782 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:17.172220 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:17.173737 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:17.174533 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:17.174737 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:17.179586 (+ 4849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:17.179600 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:17.179618 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:30:17.179673 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:17.183080 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:17.183939 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:17.183949 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:17.183957 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:17.184154 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:17.184594 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:17.184599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:17.184601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:17.185251 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:17.185583 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:17.186108 (+ 525us) server_negotiation.cc:299] Negotiation successful
0810 23:30:17.186585 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":377,"threads_started":1}
I20260810 23:30:17.194551 10289 raft_consensus.cc:348] T 00000000000000000000000000000000 P 195a406933e84fd9b05a406ee8848425 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195a406933e84fd9b05a406ee8848425" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37827 } }
I20260810 23:30:17.195025 10289 raft_consensus.cc:374] T 00000000000000000000000000000000 P 195a406933e84fd9b05a406ee8848425 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:30:17.195127 10289 raft_consensus.cc:729] T 00000000000000000000000000000000 P 195a406933e84fd9b05a406ee8848425 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195a406933e84fd9b05a406ee8848425, State: Initialized, Role: FOLLOWER
I20260810 23:30:17.195889 10289 consensus_queue.cc:260] T 00000000000000000000000000000000 P 195a406933e84fd9b05a406ee8848425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195a406933e84fd9b05a406ee8848425" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37827 } }
I20260810 23:30:17.196175 10289 raft_consensus.cc:388] T 00000000000000000000000000000000 P 195a406933e84fd9b05a406ee8848425 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:30:17.196332 10289 raft_consensus.cc:482] T 00000000000000000000000000000000 P 195a406933e84fd9b05a406ee8848425 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:30:17.196521 10289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 195a406933e84fd9b05a406ee8848425 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:30:17.199028 10289 raft_consensus.cc:504] T 00000000000000000000000000000000 P 195a406933e84fd9b05a406ee8848425 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195a406933e84fd9b05a406ee8848425" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37827 } }
I20260810 23:30:17.199554 10289 leader_election.cc:302] T 00000000000000000000000000000000 P 195a406933e84fd9b05a406ee8848425 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 195a406933e84fd9b05a406ee8848425; no voters:
I20260810 23:30:17.200235 10289 leader_election.cc:288] T 00000000000000000000000000000000 P 195a406933e84fd9b05a406ee8848425 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:30:17.200399 10294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 195a406933e84fd9b05a406ee8848425 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:30:17.201669 10294 raft_consensus.cc:686] T 00000000000000000000000000000000 P 195a406933e84fd9b05a406ee8848425 [term 1 LEADER]: Becoming Leader. State: Replica: 195a406933e84fd9b05a406ee8848425, State: Running, Role: LEADER
I20260810 23:30:17.202299 10294 consensus_queue.cc:237] T 00000000000000000000000000000000 P 195a406933e84fd9b05a406ee8848425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195a406933e84fd9b05a406ee8848425" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37827 } }
I20260810 23:30:17.202592 10289 sys_catalog.cc:564] T 00000000000000000000000000000000 P 195a406933e84fd9b05a406ee8848425 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:30:17.205106 10296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 195a406933e84fd9b05a406ee8848425 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 195a406933e84fd9b05a406ee8848425. Latest consensus state: current_term: 1 leader_uuid: "195a406933e84fd9b05a406ee8848425" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195a406933e84fd9b05a406ee8848425" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37827 } } }
I20260810 23:30:17.205570 10296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 195a406933e84fd9b05a406ee8848425 [sys.catalog]: This master's current role is: LEADER
I20260810 23:30:17.206061 10295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 195a406933e84fd9b05a406ee8848425 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "195a406933e84fd9b05a406ee8848425" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195a406933e84fd9b05a406ee8848425" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37827 } } }
I20260810 23:30:17.206372 10295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 195a406933e84fd9b05a406ee8848425 [sys.catalog]: This master's current role is: LEADER
I20260810 23:30:17.208459 10302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:30:17.214294 10302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:30:17.225198 10302 catalog_manager.cc:1347] Generated new cluster ID: b9f02caa7fa1409089646bc79a5f8f76
I20260810 23:30:17.225337 10302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:30:17.255591 10302 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:30:17.257040 10302 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:30:17.269294 10302 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 195a406933e84fd9b05a406ee8848425: Generated new TSK 0
I20260810 23:30:17.270237 10302 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
I20260810 23:30:17.353462 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.65:0
--local_ip_for_outbound_sockets=127.14.175.65
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:37827
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:46307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:30:17.718773 10326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:30:17.719108 10326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:30:17.719249 10326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:30:17.726940 10326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:30:17.727087 10326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:30:17.727248 10326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.65
I20260810 23:30:17.737861 10326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:46307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:37827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.65
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:30:17.739494 10326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:30:17.741089 10326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:30:17.755390 10336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:17.755462 10334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:17.755489 10338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:30:17.755995 10326 server_base.cc:1034] running on GCE node
I20260810 23:30:17.756783 10326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:30:17.757894 10326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:30:17.759132 10326 hybrid_clock.cc:648] HybridClock initialized: now 1786404617759069 us; error 58 us; skew 500 ppm
I20260810 23:30:17.759481 10326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:30:17 dist-test-slave-rmhg krb5kdc[10170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404617, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:30:17.762928 10326 init.cc:377] Logged in from keytab as kudu/127.14.175.65@KRBTEST.COM (short username kudu)
I20260810 23:30:17.764621 10326 webserver.cc:460] Webserver started at http://127.14.175.65:39311/ using document root <none> and password file <none>
I20260810 23:30:17.765439 10326 fs_manager.cc:359] Metadata directory not provided
I20260810 23:30:17.765553 10326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:30:17.765861 10326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:30:17.769220 10326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-0/data/instance:
uuid: "4e2f30c21c6042098e3848ed578852d8"
format_stamp: "Formatted at 2026-08-10 23:30:17 on dist-test-slave-rmhg"
server_key: "61de311b7caf907261f6350556123a2e"
server_key_iv: "8ae3b9a8da82c69d9881a35669749ea3"
server_key_version: "encryptionkey@0"
I20260810 23:30:17.770061 10326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance:
uuid: "4e2f30c21c6042098e3848ed578852d8"
format_stamp: "Formatted at 2026-08-10 23:30:17 on dist-test-slave-rmhg"
server_key: "61de311b7caf907261f6350556123a2e"
server_key_iv: "8ae3b9a8da82c69d9881a35669749ea3"
server_key_version: "encryptionkey@0"
I20260810 23:30:17.776441 10326 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260810 23:30:17.780755 10348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:17.782507 10326 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:30:17.782721 10326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-0/wal
uuid: "4e2f30c21c6042098e3848ed578852d8"
format_stamp: "Formatted at 2026-08-10 23:30:17 on dist-test-slave-rmhg"
server_key: "61de311b7caf907261f6350556123a2e"
server_key_iv: "8ae3b9a8da82c69d9881a35669749ea3"
server_key_version: "encryptionkey@0"
I20260810 23:30:17.782936 10326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:30:17.799017 10326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:30:17.804257 10326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:30:17.804577 10326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:30:17.805691 10326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:30:17.807165 10326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:30:17.807247 10326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:17.807346 10326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:30:17.807396 10326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:17.840071 10326 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.65:40865
I20260810 23:30:17.840111 10530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.65:40865 every 8 connection(s)
I20260810 23:30:17.841526 10326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
I20260810 23:30:17.847304 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 10326
I20260810 23:30:17.847525 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance
I20260810 23:30:17.848182 15037 external_mini_cluster.cc:1428] Setting key 4bf41b315685ba584bdc1f2f7c381004
Aug 10 23:30:17 dist-test-slave-rmhg krb5kdc[10170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404617, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:30:17.863531 10535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:17.844367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:45171 (local address 127.14.175.126:37827)
0810 23:30:17.844858 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:17.844863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:17.845910 (+ 1047us) server_negotiation.cc:407] Connection header received
0810 23:30:17.847966 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:17.847973 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:17.848061 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:17.848163 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:17.850080 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:17.851166 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:17.852860 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:17.853452 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:17.853575 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:17.857125 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:17.857132 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:17.857137 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:30:17.857212 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:17.860210 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:17.861049 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:17.861066 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:17.861070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:17.861267 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:17.861703 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:17.861708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:17.861711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:17.862050 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:17.862187 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:17.862945 (+ 758us) server_negotiation.cc:299] Negotiation successful
0810 23:30:17.863197 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":264,"threads_started":1}
I20260810 23:30:17.864743 10537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:17.844900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:37827 (local address 127.14.175.65:45171)
0810 23:30:17.845732 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:17.845774 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:17.847640 (+ 1866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:17.848444 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:17.848454 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:17.848997 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:30:17.849854 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:17.849882 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:17.851365 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:17.851370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:17.852690 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:17.852705 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:17.853666 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:17.853672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:17.853898 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:17.854585 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:30:17.854608 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:30:17.856861 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:30:17.860391 (+ 3530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:30:17.860400 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:30:17.860417 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:30:17.860913 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:30:17.861377 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:30:17.861381 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:30:17.861383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:30:17.861594 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:30:17.862166 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:30:17.862173 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:30:17.862549 (+ 376us) client_negotiation.cc:769] Sending connection context
0810 23:30:17.862799 (+ 250us) client_negotiation.cc:240] Negotiation successful
0810 23:30:17.863257 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":356,"threads_started":1}
I20260810 23:30:17.866628 10532 heartbeater.cc:347] Connected to a master server at 127.14.175.126:37827
I20260810 23:30:17.866968 10532 heartbeater.cc:464] Registering TS with master...
I20260810 23:30:17.867801 10532 heartbeater.cc:511] Master 127.14.175.126:37827 requested a full tablet report, sending...
I20260810 23:30:17.870322 10241 ts_manager.cc:194] Registered new tserver with Master: 4e2f30c21c6042098e3848ed578852d8 (127.14.175.65:40865)
I20260810 23:30:17.872469 10241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.65@KRBTEST.COM'} at 127.14.175.65:45171
WARNING: no policy specified for kudu/127.14.175.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
I20260810 23:30:17.920033 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.66:0
--local_ip_for_outbound_sockets=127.14.175.66
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:37827
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:46307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:30:18.272066 10540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:30:18.272374 10540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:30:18.272485 10540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:30:18.279965 10540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:30:18.280117 10540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:30:18.280272 10540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.66
I20260810 23:30:18.290992 10540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:46307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:37827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.66
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:30:18.292331 10540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:30:18.293776 10540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:30:18.307651 10556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:18.307682 10553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:18.308279 10559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:30:18.308534 10540 server_base.cc:1034] running on GCE node
I20260810 23:30:18.309243 10540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:30:18.310339 10540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:30:18.311612 10540 hybrid_clock.cc:648] HybridClock initialized: now 1786404618311531 us; error 74 us; skew 500 ppm
I20260810 23:30:18.311960 10540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:30:18 dist-test-slave-rmhg krb5kdc[10170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404618, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:30:18.315222 10540 init.cc:377] Logged in from keytab as kudu/127.14.175.66@KRBTEST.COM (short username kudu)
I20260810 23:30:18.316834 10540 webserver.cc:460] Webserver started at http://127.14.175.66:34271/ using document root <none> and password file <none>
I20260810 23:30:18.317629 10540 fs_manager.cc:359] Metadata directory not provided
I20260810 23:30:18.317755 10540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:30:18.318075 10540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:30:18.321136 10540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-1/data/instance:
uuid: "00f3e08da9994b97bb97919a3e001143"
format_stamp: "Formatted at 2026-08-10 23:30:18 on dist-test-slave-rmhg"
server_key: "5bb40637ad330fc53b759a3c62fc83f9"
server_key_iv: "bde595a805940269971f702f61934d2f"
server_key_version: "encryptionkey@0"
I20260810 23:30:18.322014 10540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance:
uuid: "00f3e08da9994b97bb97919a3e001143"
format_stamp: "Formatted at 2026-08-10 23:30:18 on dist-test-slave-rmhg"
server_key: "5bb40637ad330fc53b759a3c62fc83f9"
server_key_iv: "bde595a805940269971f702f61934d2f"
server_key_version: "encryptionkey@0"
I20260810 23:30:18.327535 10540 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:30:18.331442 10570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:18.332973 10540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:30:18.333192 10540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-1/wal
uuid: "00f3e08da9994b97bb97919a3e001143"
format_stamp: "Formatted at 2026-08-10 23:30:18 on dist-test-slave-rmhg"
server_key: "5bb40637ad330fc53b759a3c62fc83f9"
server_key_iv: "bde595a805940269971f702f61934d2f"
server_key_version: "encryptionkey@0"
I20260810 23:30:18.333423 10540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:30:18.354223 10540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:30:18.358361 10540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:30:18.358668 10540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:30:18.359683 10540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:30:18.361181 10540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:30:18.361279 10540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:18.361405 10540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:30:18.361460 10540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:18.394518 10540 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.66:44533
I20260810 23:30:18.394569 10753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.66:44533 every 8 connection(s)
I20260810 23:30:18.395866 10540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
I20260810 23:30:18.401114 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 10540
I20260810 23:30:18.401317 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance
I20260810 23:30:18.401757 15037 external_mini_cluster.cc:1428] Setting key 719e2c1d871925ef115fb01648d6a9d3
Aug 10 23:30:18 dist-test-slave-rmhg krb5kdc[10170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404618, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:30:18.417491 10758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:18.398498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:37815 (local address 127.14.175.126:37827)
0810 23:30:18.398911 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:18.398916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:18.399914 (+ 998us) server_negotiation.cc:407] Connection header received
0810 23:30:18.401880 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:18.401886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:18.401976 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:18.402106 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:18.403769 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:18.404532 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:18.406783 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:18.407201 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:18.407295 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:18.411439 (+ 4144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:18.411444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:18.411447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:30:18.411482 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:18.414102 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:18.414966 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:18.414971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:18.414974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:18.415112 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:18.415567 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:18.415570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:18.415572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:18.415895 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:18.416048 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:18.416823 (+ 775us) server_negotiation.cc:299] Negotiation successful
0810 23:30:18.417077 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":196,"threads_started":1}
I20260810 23:30:18.418535 10760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:18.399025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:37827 (local address 127.14.175.66:37815)
0810 23:30:18.399737 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:18.399790 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:18.401586 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:18.402347 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:18.402359 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:18.402922 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:30:18.403596 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:18.403617 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:18.404692 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:18.404697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:18.406648 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:18.406665 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:18.407429 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:18.407436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:18.407730 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:18.408424 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:30:18.408453 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:30:18.411197 (+ 2744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:30:18.414257 (+ 3060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:30:18.414267 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:30:18.414284 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:30:18.414853 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:30:18.415216 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:30:18.415221 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:30:18.415224 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:30:18.415473 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:30:18.416034 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:30:18.416065 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:30:18.416428 (+ 363us) client_negotiation.cc:769] Sending connection context
0810 23:30:18.416685 (+ 257us) client_negotiation.cc:240] Negotiation successful
0810 23:30:18.417348 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":264,"threads_started":1}
I20260810 23:30:18.420156 10754 heartbeater.cc:347] Connected to a master server at 127.14.175.126:37827
I20260810 23:30:18.420466 10754 heartbeater.cc:464] Registering TS with master...
I20260810 23:30:18.421240 10754 heartbeater.cc:511] Master 127.14.175.126:37827 requested a full tablet report, sending...
I20260810 23:30:18.422706 10241 ts_manager.cc:194] Registered new tserver with Master: 00f3e08da9994b97bb97919a3e001143 (127.14.175.66:44533)
I20260810 23:30:18.423513 10241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.66@KRBTEST.COM'} at 127.14.175.66:37815
WARNING: no policy specified for kudu/127.14.175.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
I20260810 23:30:18.469033 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.67:0
--local_ip_for_outbound_sockets=127.14.175.67
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:37827
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:46307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:30:18.823783 10766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:30:18.824117 10766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:30:18.824276 10766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:30:18.831774 10766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:30:18.831924 10766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:30:18.832091 10766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.67
I20260810 23:30:18.844098 10766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:46307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:37827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.67
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:30:18.845755 10766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:30:18.847215 10766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:30:18.860580 10776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:18.860543 10783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:18.860507 10779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:30:18.861235 10766 server_base.cc:1034] running on GCE node
I20260810 23:30:18.861814 10766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:30:18.862746 10766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:30:18.863952 10766 hybrid_clock.cc:648] HybridClock initialized: now 1786404618863892 us; error 51 us; skew 500 ppm
I20260810 23:30:18.864264 10766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:30:18 dist-test-slave-rmhg krb5kdc[10170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404618, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:30:18.867419 10766 init.cc:377] Logged in from keytab as kudu/127.14.175.67@KRBTEST.COM (short username kudu)
I20260810 23:30:18.868888 10766 webserver.cc:460] Webserver started at http://127.14.175.67:37923/ using document root <none> and password file <none>
I20260810 23:30:18.869606 10766 fs_manager.cc:359] Metadata directory not provided
I20260810 23:30:18.869732 10766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:30:18.870034 10766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:30:18.872991 10766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-2/data/instance:
uuid: "145e5feda83c43609f08625a10a3a777"
format_stamp: "Formatted at 2026-08-10 23:30:18 on dist-test-slave-rmhg"
server_key: "2f2affa85aa779a0339216e5baab6e5d"
server_key_iv: "b968d27ba9fadfde1e3a5b9e1fc6b425"
server_key_version: "encryptionkey@0"
I20260810 23:30:18.873833 10766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance:
uuid: "145e5feda83c43609f08625a10a3a777"
format_stamp: "Formatted at 2026-08-10 23:30:18 on dist-test-slave-rmhg"
server_key: "2f2affa85aa779a0339216e5baab6e5d"
server_key_iv: "b968d27ba9fadfde1e3a5b9e1fc6b425"
server_key_version: "encryptionkey@0"
I20260810 23:30:18.875952 10532 heartbeater.cc:503] Master 127.14.175.126:37827 was elected leader, sending a full tablet report...
I20260810 23:30:18.879433 10766 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:30:18.883034 10794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:18.884431 10766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:30:18.884611 10766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-2/wal
uuid: "145e5feda83c43609f08625a10a3a777"
format_stamp: "Formatted at 2026-08-10 23:30:18 on dist-test-slave-rmhg"
server_key: "2f2affa85aa779a0339216e5baab6e5d"
server_key_iv: "b968d27ba9fadfde1e3a5b9e1fc6b425"
server_key_version: "encryptionkey@0"
I20260810 23:30:18.884810 10766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:30:18.904856 10766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:30:18.909317 10766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:30:18.909612 10766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:30:18.910712 10766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:30:18.912119 10766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:30:18.912200 10766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:18.912329 10766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:30:18.912384 10766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:18.943360 10766 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.67:40829
I20260810 23:30:18.943405 10963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.67:40829 every 8 connection(s)
I20260810 23:30:18.944715 10766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
I20260810 23:30:18.951040 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 10766
I20260810 23:30:18.951256 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance
I20260810 23:30:18.951979 15037 external_mini_cluster.cc:1428] Setting key 0500d582708d538a19b83ccf90814477
Aug 10 23:30:18 dist-test-slave-rmhg krb5kdc[10170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404618, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:30:18.965417 10970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:18.947325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:44459 (local address 127.14.175.126:37827)
0810 23:30:18.947814 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:18.947821 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:18.948817 (+ 996us) server_negotiation.cc:407] Connection header received
0810 23:30:18.950758 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:18.950763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:18.950837 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:18.950929 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:18.953193 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:18.954196 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:18.956140 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:18.956561 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:18.956662 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:18.960059 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:18.960064 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:18.960068 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:30:18.960104 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:18.962603 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:18.963334 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:18.963338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:18.963340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:18.963458 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:18.963813 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:18.963816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:18.963818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:18.964076 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:18.964249 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:18.964889 (+ 640us) server_negotiation.cc:299] Negotiation successful
0810 23:30:18.965073 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":220,"threads_started":1}
I20260810 23:30:18.966598 10971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:18.947890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:37827 (local address 127.14.175.67:44459)
0810 23:30:18.948637 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:18.948688 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:18.950430 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:18.951270 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:18.951285 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:18.952093 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:30:18.952937 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:18.952966 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:18.954396 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:18.954403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:18.955992 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:18.956006 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:18.956751 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:18.956756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:18.956966 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:18.957625 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:30:18.957660 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:30:18.959832 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:30:18.962746 (+ 2914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:30:18.962755 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:30:18.962771 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:30:18.963239 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:30:18.963539 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:30:18.963542 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:30:18.963544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:30:18.963734 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:30:18.964191 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:30:18.964197 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:30:18.964558 (+ 361us) client_negotiation.cc:769] Sending connection context
0810 23:30:18.964811 (+ 253us) client_negotiation.cc:240] Negotiation successful
0810 23:30:18.965286 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":283,"threads_started":1}
I20260810 23:30:18.968173 10966 heartbeater.cc:347] Connected to a master server at 127.14.175.126:37827
I20260810 23:30:18.968457 10966 heartbeater.cc:464] Registering TS with master...
I20260810 23:30:18.969223 10966 heartbeater.cc:511] Master 127.14.175.126:37827 requested a full tablet report, sending...
I20260810 23:30:18.970729 10240 ts_manager.cc:194] Registered new tserver with Master: 145e5feda83c43609f08625a10a3a777 (127.14.175.67:40829)
I20260810 23:30:18.971634 10240 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.67@KRBTEST.COM'} at 127.14.175.67:44459
I20260810 23:30:18.980492 15037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:30:18.998042 10970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:18.985391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60102 (local address 127.14.175.126:37827)
0810 23:30:18.985590 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:18.985597 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:18.985999 (+ 402us) server_negotiation.cc:407] Connection header received
0810 23:30:18.986211 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:18.986217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:18.986287 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:18.986371 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:18.987904 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:18.988633 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:18.990007 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:18.990429 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:18.990542 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:18.992879 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:18.992885 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:18.992888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:30:18.992923 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:18.995352 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:18.996047 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:18.996054 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:18.996057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:18.996190 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:18.996556 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:18.996560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:18.996562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:18.996892 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:18.997131 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:18.997510 (+ 379us) server_negotiation.cc:299] Negotiation successful
0810 23:30:18.997738 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:30:19.002108 10240 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:30:19.009351 10240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:30:19.034689 10986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:19.022481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56288 (local address 127.14.175.65:40865)
0810 23:30:19.025448 (+ 2967us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:19.025457 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:19.025488 (+ 31us) server_negotiation.cc:407] Connection header received
0810 23:30:19.025592 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:19.025600 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:19.025878 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:19.026960 (+ 1082us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:30:19.027226 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:19.028679 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:19.031502 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:19.032985 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:19.033125 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:19.033394 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:19.034175 (+ 781us) server_negotiation.cc:299] Negotiation successful
0810 23:30:19.034353 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2795,"thread_start_us":2650,"threads_started":1}
I20260810 23:30:19.034679 10983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:19.022976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:40829 (local address 127.0.0.1:60546)
0810 23:30:19.026947 (+ 3971us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:19.026966 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:19.027066 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:19.027675 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:19.027680 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:19.027706 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:30:19.028081 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:19.028094 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:19.030162 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:19.030168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:19.031848 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:19.031863 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:19.033711 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:19.033720 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:19.033883 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:19.033916 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 23:30:19.034006 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 23:30:19.034239 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3821,"mutex_wait_us":59,"thread_start_us":980,"threads_started":1}
I20260810 23:30:19.034955 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:19.023246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46126 (local address 127.14.175.66:44533)
0810 23:30:19.025720 (+ 2474us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:19.025729 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:19.025756 (+ 27us) server_negotiation.cc:407] Connection header received
0810 23:30:19.025844 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:19.025851 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:19.026066 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:19.026271 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:30:19.027080 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:19.028443 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:19.032081 (+ 3638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:19.033564 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:19.033708 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:19.033899 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:19.034448 (+ 549us) server_negotiation.cc:299] Negotiation successful
0810 23:30:19.034631 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2310,"thread_start_us":2217,"threads_started":1}
I20260810 23:30:19.035212 10988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:19.022294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:40865 (local address 127.0.0.1:56288)
0810 23:30:19.023425 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:19.023486 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:19.023685 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:19.026133 (+ 2448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:19.026141 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:19.026173 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:30:19.026719 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:19.026749 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:19.028885 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:19.028891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:19.031319 (+ 2428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:19.031339 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:19.033555 (+ 2216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:19.033563 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:19.033799 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:19.033859 (+ 60us) client_negotiation.cc:769] Sending connection context
0810 23:30:19.034005 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 23:30:19.034838 (+ 833us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"mutex_wait_us":695,"thread_start_us":4324,"threads_started":1}
I20260810 23:30:19.035570 10995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:19.023595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60546 (local address 127.14.175.67:40829)
0810 23:30:19.026184 (+ 2589us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:19.026195 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:19.027096 (+ 901us) server_negotiation.cc:407] Connection header received
0810 23:30:19.027211 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:19.027218 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:19.027471 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:19.027684 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:30:19.028270 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:19.030010 (+ 1740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:19.032001 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:19.032917 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:19.033038 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:19.033347 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:19.035185 (+ 1838us) server_negotiation.cc:299] Negotiation successful
0810 23:30:19.035355 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1173,"thread_start_us":1009,"threads_started":1}
I20260810 23:30:19.035562 10990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:19.022684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:44533 (local address 127.0.0.1:46126)
0810 23:30:19.024234 (+ 1550us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:19.024256 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:19.024363 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:19.026288 (+ 1925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:19.026295 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:19.026325 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:30:19.026878 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:19.026893 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:19.028565 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:19.028570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:19.031349 (+ 2779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:19.031359 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:19.034055 (+ 2696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:19.034061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:19.034238 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:19.034270 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 23:30:19.034360 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 23:30:19.034489 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1430,"mutex_wait_us":102,"thread_start_us":727,"threads_started":1}
I20260810 23:30:19.039731 10662 tablet_service.cc:1467] Processing CreateTablet for tablet c165aa2c7ba649b38c1aea4e5ee3f95f (DEFAULT_TABLE table=test-table [id=a97f3144dcfc4b7999cc172f78a3ab32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:30:19.039649 10862 tablet_service.cc:1467] Processing CreateTablet for tablet c165aa2c7ba649b38c1aea4e5ee3f95f (DEFAULT_TABLE table=test-table [id=a97f3144dcfc4b7999cc172f78a3ab32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:30:19.039649 10427 tablet_service.cc:1467] Processing CreateTablet for tablet c165aa2c7ba649b38c1aea4e5ee3f95f (DEFAULT_TABLE table=test-table [id=a97f3144dcfc4b7999cc172f78a3ab32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:30:19.040978 10662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c165aa2c7ba649b38c1aea4e5ee3f95f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:30:19.041085 10427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c165aa2c7ba649b38c1aea4e5ee3f95f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:30:19.041097 10862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c165aa2c7ba649b38c1aea4e5ee3f95f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:30:19.054222 10996 tablet_bootstrap.cc:492] T c165aa2c7ba649b38c1aea4e5ee3f95f P 00f3e08da9994b97bb97919a3e001143: Bootstrap starting.
I20260810 23:30:19.056684 10998 tablet_bootstrap.cc:492] T c165aa2c7ba649b38c1aea4e5ee3f95f P 4e2f30c21c6042098e3848ed578852d8: Bootstrap starting.
I20260810 23:30:19.058187 10996 tablet_bootstrap.cc:654] T c165aa2c7ba649b38c1aea4e5ee3f95f P 00f3e08da9994b97bb97919a3e001143: Neither blocks nor log segments found. Creating new log.
I20260810 23:30:19.058322 10997 tablet_bootstrap.cc:492] T c165aa2c7ba649b38c1aea4e5ee3f95f P 145e5feda83c43609f08625a10a3a777: Bootstrap starting.
I20260810 23:30:19.059721 10996 log.cc:824] T c165aa2c7ba649b38c1aea4e5ee3f95f P 00f3e08da9994b97bb97919a3e001143: Log is configured to *not* fsync() on all Append() calls
I20260810 23:30:19.060489 10998 tablet_bootstrap.cc:654] T c165aa2c7ba649b38c1aea4e5ee3f95f P 4e2f30c21c6042098e3848ed578852d8: Neither blocks nor log segments found. Creating new log.
I20260810 23:30:19.061589 10997 tablet_bootstrap.cc:654] T c165aa2c7ba649b38c1aea4e5ee3f95f P 145e5feda83c43609f08625a10a3a777: Neither blocks nor log segments found. Creating new log.
I20260810 23:30:19.061782 10998 log.cc:824] T c165aa2c7ba649b38c1aea4e5ee3f95f P 4e2f30c21c6042098e3848ed578852d8: Log is configured to *not* fsync() on all Append() calls
I20260810 23:30:19.062933 10997 log.cc:824] T c165aa2c7ba649b38c1aea4e5ee3f95f P 145e5feda83c43609f08625a10a3a777: Log is configured to *not* fsync() on all Append() calls
I20260810 23:30:19.063398 10996 tablet_bootstrap.cc:492] T c165aa2c7ba649b38c1aea4e5ee3f95f P 00f3e08da9994b97bb97919a3e001143: No bootstrap required, opened a new log
I20260810 23:30:19.063755 10996 ts_tablet_manager.cc:1397] T c165aa2c7ba649b38c1aea4e5ee3f95f P 00f3e08da9994b97bb97919a3e001143: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260810 23:30:19.065428 10998 tablet_bootstrap.cc:492] T c165aa2c7ba649b38c1aea4e5ee3f95f P 4e2f30c21c6042098e3848ed578852d8: No bootstrap required, opened a new log
I20260810 23:30:19.065604 10997 tablet_bootstrap.cc:492] T c165aa2c7ba649b38c1aea4e5ee3f95f P 145e5feda83c43609f08625a10a3a777: No bootstrap required, opened a new log
I20260810 23:30:19.065730 10998 ts_tablet_manager.cc:1397] T c165aa2c7ba649b38c1aea4e5ee3f95f P 4e2f30c21c6042098e3848ed578852d8: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260810 23:30:19.065868 10997 ts_tablet_manager.cc:1397] T c165aa2c7ba649b38c1aea4e5ee3f95f P 145e5feda83c43609f08625a10a3a777: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260810 23:30:19.073427 10997 raft_consensus.cc:348] T c165aa2c7ba649b38c1aea4e5ee3f95f P 145e5feda83c43609f08625a10a3a777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2f30c21c6042098e3848ed578852d8" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 40865 } } peers { permanent_uuid: "00f3e08da9994b97bb97919a3e001143" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44533 } } peers { permanent_uuid: "145e5feda83c43609f08625a10a3a777" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40829 } }
I20260810 23:30:19.073987 10997 raft_consensus.cc:374] T c165aa2c7ba649b38c1aea4e5ee3f95f P 145e5feda83c43609f08625a10a3a777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:30:19.074136 10997 raft_consensus.cc:729] T c165aa2c7ba649b38c1aea4e5ee3f95f P 145e5feda83c43609f08625a10a3a777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 145e5feda83c43609f08625a10a3a777, State: Initialized, Role: FOLLOWER
I20260810 23:30:19.074810 10997 consensus_queue.cc:260] T c165aa2c7ba649b38c1aea4e5ee3f95f P 145e5feda83c43609f08625a10a3a777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2f30c21c6042098e3848ed578852d8" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 40865 } } peers { permanent_uuid: "00f3e08da9994b97bb97919a3e001143" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44533 } } peers { permanent_uuid: "145e5feda83c43609f08625a10a3a777" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40829 } }
I20260810 23:30:19.074658 10996 raft_consensus.cc:348] T c165aa2c7ba649b38c1aea4e5ee3f95f P 00f3e08da9994b97bb97919a3e001143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2f30c21c6042098e3848ed578852d8" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 40865 } } peers { permanent_uuid: "00f3e08da9994b97bb97919a3e001143" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44533 } } peers { permanent_uuid: "145e5feda83c43609f08625a10a3a777" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40829 } }
I20260810 23:30:19.075105 10996 raft_consensus.cc:374] T c165aa2c7ba649b38c1aea4e5ee3f95f P 00f3e08da9994b97bb97919a3e001143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:30:19.075218 10996 raft_consensus.cc:729] T c165aa2c7ba649b38c1aea4e5ee3f95f P 00f3e08da9994b97bb97919a3e001143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00f3e08da9994b97bb97919a3e001143, State: Initialized, Role: FOLLOWER
I20260810 23:30:19.075978 10966 heartbeater.cc:503] Master 127.14.175.126:37827 was elected leader, sending a full tablet report...
I20260810 23:30:19.076048 10996 consensus_queue.cc:260] T c165aa2c7ba649b38c1aea4e5ee3f95f P 00f3e08da9994b97bb97919a3e001143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2f30c21c6042098e3848ed578852d8" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 40865 } } peers { permanent_uuid: "00f3e08da9994b97bb97919a3e001143" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44533 } } peers { permanent_uuid: "145e5feda83c43609f08625a10a3a777" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40829 } }
I20260810 23:30:19.076617 10997 ts_tablet_manager.cc:1428] T c165aa2c7ba649b38c1aea4e5ee3f95f P 145e5feda83c43609f08625a10a3a777: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260810 23:30:19.077250 10754 heartbeater.cc:503] Master 127.14.175.126:37827 was elected leader, sending a full tablet report...
I20260810 23:30:19.077819 10998 raft_consensus.cc:348] T c165aa2c7ba649b38c1aea4e5ee3f95f P 4e2f30c21c6042098e3848ed578852d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2f30c21c6042098e3848ed578852d8" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 40865 } } peers { permanent_uuid: "00f3e08da9994b97bb97919a3e001143" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44533 } } peers { permanent_uuid: "145e5feda83c43609f08625a10a3a777" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40829 } }
I20260810 23:30:19.078222 10998 raft_consensus.cc:374] T c165aa2c7ba649b38c1aea4e5ee3f95f P 4e2f30c21c6042098e3848ed578852d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:30:19.078329 10998 raft_consensus.cc:729] T c165aa2c7ba649b38c1aea4e5ee3f95f P 4e2f30c21c6042098e3848ed578852d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e2f30c21c6042098e3848ed578852d8, State: Initialized, Role: FOLLOWER
I20260810 23:30:19.078945 10998 consensus_queue.cc:260] T c165aa2c7ba649b38c1aea4e5ee3f95f P 4e2f30c21c6042098e3848ed578852d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2f30c21c6042098e3848ed578852d8" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 40865 } } peers { permanent_uuid: "00f3e08da9994b97bb97919a3e001143" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44533 } } peers { permanent_uuid: "145e5feda83c43609f08625a10a3a777" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40829 } }
I20260810 23:30:19.079459 10996 ts_tablet_manager.cc:1428] T c165aa2c7ba649b38c1aea4e5ee3f95f P 00f3e08da9994b97bb97919a3e001143: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
I20260810 23:30:19.080858 10998 ts_tablet_manager.cc:1428] T c165aa2c7ba649b38c1aea4e5ee3f95f P 4e2f30c21c6042098e3848ed578852d8: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
W20260810 23:30:19.094184 10534 tablet.cc:2369] T c165aa2c7ba649b38c1aea4e5ee3f95f P 4e2f30c21c6042098e3848ed578852d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:30:19.099258 11006 raft_consensus.cc:482] T c165aa2c7ba649b38c1aea4e5ee3f95f P 00f3e08da9994b97bb97919a3e001143 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:30:19.099591 11006 raft_consensus.cc:504] T c165aa2c7ba649b38c1aea4e5ee3f95f P 00f3e08da9994b97bb97919a3e001143 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2f30c21c6042098e3848ed578852d8" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 40865 } } peers { permanent_uuid: "00f3e08da9994b97bb97919a3e001143" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44533 } } peers { permanent_uuid: "145e5feda83c43609f08625a10a3a777" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40829 } }
I20260810 23:30:19.101442 11006 leader_election.cc:288] T c165aa2c7ba649b38c1aea4e5ee3f95f P 00f3e08da9994b97bb97919a3e001143 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e2f30c21c6042098e3848ed578852d8 (127.14.175.65:40865), 145e5feda83c43609f08625a10a3a777 (127.14.175.67:40829)
I20260810 23:30:19.111030 10995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:19.102044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:60179 (local address 127.14.175.67:40829)
0810 23:30:19.102251 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:19.102258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:19.102936 (+ 678us) server_negotiation.cc:407] Connection header received
0810 23:30:19.103264 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:19.103268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:19.103330 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:19.103377 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:30:19.104265 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:19.105439 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:19.108595 (+ 3156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:19.109883 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:19.110032 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:19.110111 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:19.110687 (+ 576us) server_negotiation.cc:299] Negotiation successful
0810 23:30:19.110816 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 23:30:19.111001 10986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:19.102360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:40149 (local address 127.14.175.65:40865)
0810 23:30:19.102559 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:19.102565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:19.102936 (+ 371us) server_negotiation.cc:407] Connection header received
0810 23:30:19.103274 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:19.103280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:19.103358 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:19.103449 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:30:19.104272 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:19.105439 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:19.108595 (+ 3156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:19.109849 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:19.110032 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:19.110111 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:19.110617 (+ 506us) server_negotiation.cc:299] Negotiation successful
0810 23:30:19.110753 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 23:30:19.111230 11009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:19.101716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:40829 (local address 127.14.175.66:60179)
0810 23:30:19.102817 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:19.102840 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:19.103063 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:19.103571 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:19.103575 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:19.103597 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:30:19.104073 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:19.104092 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:19.105607 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:19.105612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:19.108414 (+ 2802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:19.108434 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:19.110124 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:19.110131 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:19.110409 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:19.110483 (+ 74us) client_negotiation.cc:769] Sending connection context
0810 23:30:19.110653 (+ 170us) client_negotiation.cc:240] Negotiation successful
0810 23:30:19.110760 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"mutex_wait_us":237,"thread_start_us":1128,"threads_started":1}
I20260810 23:30:19.111269 11010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:19.101820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:40865 (local address 127.14.175.66:40149)
0810 23:30:19.102826 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:19.102840 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:19.103096 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:19.103499 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:19.103503 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:19.103531 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:30:19.104107 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:19.104116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:19.105604 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:19.105610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:19.108419 (+ 2809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:19.108434 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:19.110125 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:19.110131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:19.110405 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:19.110487 (+ 82us) client_negotiation.cc:769] Sending connection context
0810 23:30:19.110589 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 23:30:19.110740 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":899,"mutex_wait_us":244,"spinlock_wait_cycles":23296,"thread_start_us":751,"threads_started":1}
I20260810 23:30:19.112396 10456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c165aa2c7ba649b38c1aea4e5ee3f95f" candidate_uuid: "00f3e08da9994b97bb97919a3e001143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e2f30c21c6042098e3848ed578852d8" is_pre_election: true
I20260810 23:30:19.112445 10886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c165aa2c7ba649b38c1aea4e5ee3f95f" candidate_uuid: "00f3e08da9994b97bb97919a3e001143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "145e5feda83c43609f08625a10a3a777" is_pre_election: true
I20260810 23:30:19.112941 10886 raft_consensus.cc:2454] T c165aa2c7ba649b38c1aea4e5ee3f95f P 145e5feda83c43609f08625a10a3a777 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00f3e08da9994b97bb97919a3e001143 in term 0.
I20260810 23:30:19.112942 10456 raft_consensus.cc:2454] T c165aa2c7ba649b38c1aea4e5ee3f95f P 4e2f30c21c6042098e3848ed578852d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00f3e08da9994b97bb97919a3e001143 in term 0.
I20260810 23:30:19.113649 10580 leader_election.cc:302] T c165aa2c7ba649b38c1aea4e5ee3f95f P 00f3e08da9994b97bb97919a3e001143 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00f3e08da9994b97bb97919a3e001143, 4e2f30c21c6042098e3848ed578852d8; no voters:
I20260810 23:30:19.114066 11006 raft_consensus.cc:2781] T c165aa2c7ba649b38c1aea4e5ee3f95f P 00f3e08da9994b97bb97919a3e001143 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:30:19.114259 11006 raft_consensus.cc:482] T c165aa2c7ba649b38c1aea4e5ee3f95f P 00f3e08da9994b97bb97919a3e001143 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:30:19.114378 11006 raft_consensus.cc:3037] T c165aa2c7ba649b38c1aea4e5ee3f95f P 00f3e08da9994b97bb97919a3e001143 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:30:19.117759 11006 raft_consensus.cc:504] T c165aa2c7ba649b38c1aea4e5ee3f95f P 00f3e08da9994b97bb97919a3e001143 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2f30c21c6042098e3848ed578852d8" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 40865 } } peers { permanent_uuid: "00f3e08da9994b97bb97919a3e001143" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44533 } } peers { permanent_uuid: "145e5feda83c43609f08625a10a3a777" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40829 } }
I20260810 23:30:19.118629 11006 leader_election.cc:288] T c165aa2c7ba649b38c1aea4e5ee3f95f P 00f3e08da9994b97bb97919a3e001143 [CANDIDATE]: Term 1 election: Requested vote from peers 4e2f30c21c6042098e3848ed578852d8 (127.14.175.65:40865), 145e5feda83c43609f08625a10a3a777 (127.14.175.67:40829)
I20260810 23:30:19.119310 10886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c165aa2c7ba649b38c1aea4e5ee3f95f" candidate_uuid: "00f3e08da9994b97bb97919a3e001143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "145e5feda83c43609f08625a10a3a777"
I20260810 23:30:19.119585 10886 raft_consensus.cc:3037] T c165aa2c7ba649b38c1aea4e5ee3f95f P 145e5feda83c43609f08625a10a3a777 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:30:19.121502 10456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c165aa2c7ba649b38c1aea4e5ee3f95f" candidate_uuid: "00f3e08da9994b97bb97919a3e001143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e2f30c21c6042098e3848ed578852d8"
I20260810 23:30:19.121765 10456 raft_consensus.cc:3037] T c165aa2c7ba649b38c1aea4e5ee3f95f P 4e2f30c21c6042098e3848ed578852d8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:30:19.122499 10886 raft_consensus.cc:2454] T c165aa2c7ba649b38c1aea4e5ee3f95f P 145e5feda83c43609f08625a10a3a777 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00f3e08da9994b97bb97919a3e001143 in term 1.
I20260810 23:30:19.123205 10574 leader_election.cc:302] T c165aa2c7ba649b38c1aea4e5ee3f95f P 00f3e08da9994b97bb97919a3e001143 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00f3e08da9994b97bb97919a3e001143, 145e5feda83c43609f08625a10a3a777; no voters:
I20260810 23:30:19.123663 11006 raft_consensus.cc:2781] T c165aa2c7ba649b38c1aea4e5ee3f95f P 00f3e08da9994b97bb97919a3e001143 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:30:19.124225 11006 raft_consensus.cc:686] T c165aa2c7ba649b38c1aea4e5ee3f95f P 00f3e08da9994b97bb97919a3e001143 [term 1 LEADER]: Becoming Leader. State: Replica: 00f3e08da9994b97bb97919a3e001143, State: Running, Role: LEADER
I20260810 23:30:19.124660 10456 raft_consensus.cc:2454] T c165aa2c7ba649b38c1aea4e5ee3f95f P 4e2f30c21c6042098e3848ed578852d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00f3e08da9994b97bb97919a3e001143 in term 1.
I20260810 23:30:19.124781 11006 consensus_queue.cc:237] T c165aa2c7ba649b38c1aea4e5ee3f95f P 00f3e08da9994b97bb97919a3e001143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2f30c21c6042098e3848ed578852d8" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 40865 } } peers { permanent_uuid: "00f3e08da9994b97bb97919a3e001143" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44533 } } peers { permanent_uuid: "145e5feda83c43609f08625a10a3a777" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40829 } }
I20260810 23:30:19.130378 10238 catalog_manager.cc:5495] T c165aa2c7ba649b38c1aea4e5ee3f95f P 00f3e08da9994b97bb97919a3e001143 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00f3e08da9994b97bb97919a3e001143 (127.14.175.66). New cstate: current_term: 1 leader_uuid: "00f3e08da9994b97bb97919a3e001143" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2f30c21c6042098e3848ed578852d8" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 40865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00f3e08da9994b97bb97919a3e001143" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 44533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "145e5feda83c43609f08625a10a3a777" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40829 } health_report { overall_health: UNKNOWN } } }
W20260810 23:30:19.147936 10756 tablet.cc:2369] T c165aa2c7ba649b38c1aea4e5ee3f95f P 00f3e08da9994b97bb97919a3e001143: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:30:19.162346 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:19.155788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46128 (local address 127.14.175.66:44533)
0810 23:30:19.155971 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:19.155976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:19.156031 (+ 55us) server_negotiation.cc:407] Connection header received
0810 23:30:19.156147 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:19.156151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:19.156235 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:19.156366 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:30:19.156876 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:19.157690 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:19.159765 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:19.160211 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:19.160319 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:19.160964 (+ 645us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:30:19.161599 (+ 635us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:30:19.161717 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:19.161892 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 23:30:19.162027 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:30:19.171785 10456 raft_consensus.cc:1261] T c165aa2c7ba649b38c1aea4e5ee3f95f P 4e2f30c21c6042098e3848ed578852d8 [term 1 FOLLOWER]: Refusing update from remote peer 00f3e08da9994b97bb97919a3e001143: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:30:19.171785 10886 raft_consensus.cc:1261] T c165aa2c7ba649b38c1aea4e5ee3f95f P 145e5feda83c43609f08625a10a3a777 [term 1 FOLLOWER]: Refusing update from remote peer 00f3e08da9994b97bb97919a3e001143: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:30:19.173209 11011 consensus_queue.cc:1035] T c165aa2c7ba649b38c1aea4e5ee3f95f P 00f3e08da9994b97bb97919a3e001143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e2f30c21c6042098e3848ed578852d8" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 40865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:30:19.173660 11006 consensus_queue.cc:1035] T c165aa2c7ba649b38c1aea4e5ee3f95f P 00f3e08da9994b97bb97919a3e001143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "145e5feda83c43609f08625a10a3a777" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:30:19.187687 11016 mvcc.cc:205] Tried to move back new op lower bound from 7317113320115118080 to 7317113319940284416. Current Snapshot: MvccSnapshot[applied={T|T < 7317113320115118080}]
I20260810 23:30:19.200981 10240 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:30:19.201364 10240 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:30:19.205710 10240 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 195a406933e84fd9b05a406ee8848425: Sending DeleteTablet for 3 replicas of tablet c165aa2c7ba649b38c1aea4e5ee3f95f
I20260810 23:30:19.207257 10427 tablet_service.cc:1514] Processing DeleteTablet for tablet c165aa2c7ba649b38c1aea4e5ee3f95f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:30:19 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:56288
I20260810 23:30:19.207263 10662 tablet_service.cc:1514] Processing DeleteTablet for tablet c165aa2c7ba649b38c1aea4e5ee3f95f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:30:19 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:46126
I20260810 23:30:19.207700 10862 tablet_service.cc:1514] Processing DeleteTablet for tablet c165aa2c7ba649b38c1aea4e5ee3f95f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:30:19 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:60546
I20260810 23:30:19.209959 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 10326
I20260810 23:30:19.219456 11028 tablet_replica.cc:330] T c165aa2c7ba649b38c1aea4e5ee3f95f P 4e2f30c21c6042098e3848ed578852d8: stopping tablet replica
I20260810 23:30:19.220279 11028 raft_consensus.cc:2229] T c165aa2c7ba649b38c1aea4e5ee3f95f P 4e2f30c21c6042098e3848ed578852d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:30:19.221642 11028 raft_consensus.cc:2258] T c165aa2c7ba649b38c1aea4e5ee3f95f P 4e2f30c21c6042098e3848ed578852d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:30:19.222295 11029 tablet_replica.cc:330] T c165aa2c7ba649b38c1aea4e5ee3f95f P 145e5feda83c43609f08625a10a3a777: stopping tablet replica
I20260810 23:30:19.222604 11027 tablet_replica.cc:330] T c165aa2c7ba649b38c1aea4e5ee3f95f P 00f3e08da9994b97bb97919a3e001143: stopping tablet replica
I20260810 23:30:19.223160 11029 raft_consensus.cc:2229] T c165aa2c7ba649b38c1aea4e5ee3f95f P 145e5feda83c43609f08625a10a3a777 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:30:19.223183 11027 raft_consensus.cc:2229] T c165aa2c7ba649b38c1aea4e5ee3f95f P 00f3e08da9994b97bb97919a3e001143 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:30:19.223954 11027 raft_consensus.cc:2258] T c165aa2c7ba649b38c1aea4e5ee3f95f P 00f3e08da9994b97bb97919a3e001143 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:30:19.225821 11028 ts_tablet_manager.cc:1905] T c165aa2c7ba649b38c1aea4e5ee3f95f P 4e2f30c21c6042098e3848ed578852d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:30:19.226116 11029 raft_consensus.cc:2258] T c165aa2c7ba649b38c1aea4e5ee3f95f P 145e5feda83c43609f08625a10a3a777 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:30:19.226632 11027 ts_tablet_manager.cc:1905] T c165aa2c7ba649b38c1aea4e5ee3f95f P 00f3e08da9994b97bb97919a3e001143: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:30:19.228434 11029 ts_tablet_manager.cc:1905] T c165aa2c7ba649b38c1aea4e5ee3f95f P 145e5feda83c43609f08625a10a3a777: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 23:30:19 dist-test-slave-rmhg krb5kdc[10170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.65@KRBTEST.COM
I20260810 23:30:19.234530 11027 ts_tablet_manager.cc:1918] T c165aa2c7ba649b38c1aea4e5ee3f95f P 00f3e08da9994b97bb97919a3e001143: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:30:19.234838 11027 log.cc:1196] T c165aa2c7ba649b38c1aea4e5ee3f95f P 00f3e08da9994b97bb97919a3e001143: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-1/wal/wals/c165aa2c7ba649b38c1aea4e5ee3f95f
I20260810 23:30:19.235548 11027 ts_tablet_manager.cc:1939] T c165aa2c7ba649b38c1aea4e5ee3f95f P 00f3e08da9994b97bb97919a3e001143: Deleting consensus metadata
I20260810 23:30:19.235741 11028 ts_tablet_manager.cc:1918] T c165aa2c7ba649b38c1aea4e5ee3f95f P 4e2f30c21c6042098e3848ed578852d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:30:19.238021 11028 log.cc:1196] T c165aa2c7ba649b38c1aea4e5ee3f95f P 4e2f30c21c6042098e3848ed578852d8: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-0/wal/wals/c165aa2c7ba649b38c1aea4e5ee3f95f
I20260810 23:30:19.238708 11028 ts_tablet_manager.cc:1939] T c165aa2c7ba649b38c1aea4e5ee3f95f P 4e2f30c21c6042098e3848ed578852d8: Deleting consensus metadata
I20260810 23:30:19.239138 10216 catalog_manager.cc:4841] TS 00f3e08da9994b97bb97919a3e001143 (127.14.175.66:44533): tablet c165aa2c7ba649b38c1aea4e5ee3f95f (table test-table [id=a97f3144dcfc4b7999cc172f78a3ab32]) successfully deleted
I20260810 23:30:19.240837 10217 catalog_manager.cc:4841] TS 4e2f30c21c6042098e3848ed578852d8 (127.14.175.65:40865): tablet c165aa2c7ba649b38c1aea4e5ee3f95f (table test-table [id=a97f3144dcfc4b7999cc172f78a3ab32]) successfully deleted
I20260810 23:30:19.243233 10986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:19.211683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56292 (local address 127.14.175.65:40865)
0810 23:30:19.211893 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:19.211900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:19.216674 (+ 4774us) server_negotiation.cc:407] Connection header received
0810 23:30:19.216857 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:19.216864 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:19.216967 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:19.217105 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:19.224814 (+ 7709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:19.225928 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:19.228082 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:19.228710 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:19.228819 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:19.234609 (+ 5790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:19.234629 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:19.234651 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:30:19.234701 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:19.238766 (+ 4065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:19.240425 (+ 1659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:19.240436 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:19.240440 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:19.240633 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:19.241138 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:19.241144 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:19.241168 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:19.241666 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:19.241999 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:19.242387 (+ 388us) server_negotiation.cc:299] Negotiation successful
0810 23:30:19.242635 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 23:30:19.243194 11029 ts_tablet_manager.cc:1918] T c165aa2c7ba649b38c1aea4e5ee3f95f P 145e5feda83c43609f08625a10a3a777: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:30:19.243379 11029 log.cc:1196] T c165aa2c7ba649b38c1aea4e5ee3f95f P 145e5feda83c43609f08625a10a3a777: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404299514076-15037-0/minicluster-data/ts-2/wal/wals/c165aa2c7ba649b38c1aea4e5ee3f95f
I20260810 23:30:19.243988 11029 ts_tablet_manager.cc:1939] T c165aa2c7ba649b38c1aea4e5ee3f95f P 145e5feda83c43609f08625a10a3a777: Deleting consensus metadata
I20260810 23:30:19.244055 10523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:30:19.246443 10210 catalog_manager.cc:4841] TS 145e5feda83c43609f08625a10a3a777 (127.14.175.67:40829): tablet c165aa2c7ba649b38c1aea4e5ee3f95f (table test-table [id=a97f3144dcfc4b7999cc172f78a3ab32]) successfully deleted
I20260810 23:30:19.335366 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 10326
I20260810 23:30:19.377967 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 10540
Aug 10 23:30:19 dist-test-slave-rmhg krb5kdc[10170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.66@KRBTEST.COM
I20260810 23:30:19.395257 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:19.378793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46136 (local address 127.14.175.66:44533)
0810 23:30:19.379034 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:19.379039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:19.379061 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:30:19.379196 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:19.379202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:19.379283 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:19.379373 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:19.380753 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:19.381933 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:19.383294 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:19.383758 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:19.383883 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:19.388581 (+ 4698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:19.388598 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:19.388620 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:30:19.388682 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:19.391929 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:19.392709 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:19.392717 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:19.392720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:19.392874 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:19.393361 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:19.393365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:19.393367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:19.393808 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:19.394265 (+ 457us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:19.394444 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 23:30:19.394688 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260810 23:30:19.396090 10749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:30:19.511161 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 10540
I20260810 23:30:19.537698 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 10766
Aug 10 23:30:19 dist-test-slave-rmhg krb5kdc[10170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.67@KRBTEST.COM
I20260810 23:30:19.552999 10995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:19.538492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60560 (local address 127.14.175.67:40829)
0810 23:30:19.538687 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:19.538692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:19.538710 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:30:19.538825 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:19.538829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:19.538892 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:19.538978 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:19.540360 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:19.541127 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:19.542481 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:19.542875 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:19.542968 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:19.547360 (+ 4392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:19.547378 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:19.547402 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:30:19.547446 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:19.550271 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:19.551058 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:19.551067 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:19.551069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:19.551236 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:19.551573 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:19.551576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:19.551578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:19.551974 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:19.552205 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:19.552491 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 23:30:19.552691 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260810 23:30:19.553782 10957 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:30:19.642764 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 10766
I20260810 23:30:19.666079 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 10191
I20260810 23:30:19.667277 10283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:30:19.781706 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 10191
2026-08-10T23:30:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4659 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T23:30:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:30:19Z Disabled control of system clock
I20260810 23:30:19.886358 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:38331
--webserver_interface=127.14.175.126
--webserver_port=0
--builtin_ntp_servers=127.14.175.84:35467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:38331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:30:20.254496 11044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:30:20.254838 11044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:30:20.254968 11044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:30:20.262441 11044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:30:20.262566 11044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:30:20.262638 11044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:30:20.262738 11044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:30:20.262794 11044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:30:20.272579 11044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:35467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:38331
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:38331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:30:20.274097 11044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:30:20.276079 11044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:30:20.288542 11052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:20.288604 11054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:20.288560 11050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:30:20.289609 11044 server_base.cc:1034] running on GCE node
I20260810 23:30:20.290462 11044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:30:20.291980 11044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:30:20.293243 11044 hybrid_clock.cc:648] HybridClock initialized: now 1786404620293190 us; error 43 us; skew 500 ppm
I20260810 23:30:20.293612 11044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:30:20.299379 11044 webserver.cc:460] Webserver started at http://127.14.175.126:40099/ using document root <none> and password file <none>
I20260810 23:30:20.300148 11044 fs_manager.cc:359] Metadata directory not provided
I20260810 23:30:20.300272 11044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:30:20.300596 11044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:30:20.303777 11044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "a2e035b11de047ec85beb4c68c77bec8"
format_stamp: "Formatted at 2026-08-10 23:30:20 on dist-test-slave-rmhg"
server_key: "00cbc6f7176dddf85f9e279b17138ec9"
server_key_iv: "4641051ed0c3e050bd70caf8eb482dcc"
server_key_version: "encryptionkey@0"
I20260810 23:30:20.304675 11044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "a2e035b11de047ec85beb4c68c77bec8"
format_stamp: "Formatted at 2026-08-10 23:30:20 on dist-test-slave-rmhg"
server_key: "00cbc6f7176dddf85f9e279b17138ec9"
server_key_iv: "4641051ed0c3e050bd70caf8eb482dcc"
server_key_version: "encryptionkey@0"
I20260810 23:30:20.310323 11044 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 23:30:20.314250 11062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:20.315645 11044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:30:20.315860 11044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "a2e035b11de047ec85beb4c68c77bec8"
format_stamp: "Formatted at 2026-08-10 23:30:20 on dist-test-slave-rmhg"
server_key: "00cbc6f7176dddf85f9e279b17138ec9"
server_key_iv: "4641051ed0c3e050bd70caf8eb482dcc"
server_key_version: "encryptionkey@0"
I20260810 23:30:20.316074 11044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:30:20.338860 11044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:30:20.339861 11044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:30:20.340186 11044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:30:20.359679 11044 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:38331
I20260810 23:30:20.359694 11141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:38331 every 8 connection(s)
I20260810 23:30:20.361193 11044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:30:20.365327 11142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:30:20.369506 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 11044
I20260810 23:30:20.369710 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:30:20.370281 15037 external_mini_cluster.cc:1428] Setting key 2ae1ecdd3d47f7d275b40db13d39a4e3
I20260810 23:30:20.380724 11142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2e035b11de047ec85beb4c68c77bec8: Bootstrap starting.
I20260810 23:30:20.385465 11142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2e035b11de047ec85beb4c68c77bec8: Neither blocks nor log segments found. Creating new log.
I20260810 23:30:20.386871 11142 log.cc:824] T 00000000000000000000000000000000 P a2e035b11de047ec85beb4c68c77bec8: Log is configured to *not* fsync() on all Append() calls
I20260810 23:30:20.388226 11147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:20.372377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49130 (local address 127.14.175.126:38331)
0810 23:30:20.373549 (+ 1172us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:20.373570 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:20.373616 (+ 46us) server_negotiation.cc:407] Connection header received
0810 23:30:20.375452 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:20.375496 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:20.376276 (+ 780us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:20.376927 (+ 651us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:20.377946 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:20.379969 (+ 2023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:20.382102 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:20.383308 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:20.383545 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:20.385415 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:20.385424 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:20.385441 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:20.386127 (+ 686us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:20.386233 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:20.386258 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:20.386352 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:20.386599 (+ 247us) server_negotiation.cc:299] Negotiation successful
0810 23:30:20.386894 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":395,"threads_started":1}
I20260810 23:30:20.390300 11142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2e035b11de047ec85beb4c68c77bec8: No bootstrap required, opened a new log
I20260810 23:30:20.398386 11142 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2e035b11de047ec85beb4c68c77bec8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e035b11de047ec85beb4c68c77bec8" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38331 } }
I20260810 23:30:20.398818 11142 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2e035b11de047ec85beb4c68c77bec8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:30:20.398932 11142 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2e035b11de047ec85beb4c68c77bec8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e035b11de047ec85beb4c68c77bec8, State: Initialized, Role: FOLLOWER
I20260810 23:30:20.399567 11142 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2e035b11de047ec85beb4c68c77bec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e035b11de047ec85beb4c68c77bec8" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38331 } }
I20260810 23:30:20.399817 11142 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2e035b11de047ec85beb4c68c77bec8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:30:20.399964 11142 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2e035b11de047ec85beb4c68c77bec8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:30:20.400133 11142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2e035b11de047ec85beb4c68c77bec8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:30:20.402470 11142 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2e035b11de047ec85beb4c68c77bec8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e035b11de047ec85beb4c68c77bec8" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38331 } }
I20260810 23:30:20.403036 11142 leader_election.cc:302] T 00000000000000000000000000000000 P a2e035b11de047ec85beb4c68c77bec8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2e035b11de047ec85beb4c68c77bec8; no voters:
I20260810 23:30:20.403681 11142 leader_election.cc:288] T 00000000000000000000000000000000 P a2e035b11de047ec85beb4c68c77bec8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:30:20.403864 11150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2e035b11de047ec85beb4c68c77bec8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:30:20.404588 11150 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2e035b11de047ec85beb4c68c77bec8 [term 1 LEADER]: Becoming Leader. State: Replica: a2e035b11de047ec85beb4c68c77bec8, State: Running, Role: LEADER
I20260810 23:30:20.405205 11150 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2e035b11de047ec85beb4c68c77bec8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e035b11de047ec85beb4c68c77bec8" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38331 } }
I20260810 23:30:20.406002 11142 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2e035b11de047ec85beb4c68c77bec8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:30:20.408393 11154 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2e035b11de047ec85beb4c68c77bec8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2e035b11de047ec85beb4c68c77bec8. Latest consensus state: current_term: 1 leader_uuid: "a2e035b11de047ec85beb4c68c77bec8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e035b11de047ec85beb4c68c77bec8" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38331 } } }
I20260810 23:30:20.408851 11154 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2e035b11de047ec85beb4c68c77bec8 [sys.catalog]: This master's current role is: LEADER
I20260810 23:30:20.409291 11151 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2e035b11de047ec85beb4c68c77bec8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2e035b11de047ec85beb4c68c77bec8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e035b11de047ec85beb4c68c77bec8" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38331 } } }
I20260810 23:30:20.409623 11151 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2e035b11de047ec85beb4c68c77bec8 [sys.catalog]: This master's current role is: LEADER
I20260810 23:30:20.410794 11161 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:30:20.415686 11161 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:30:20.426107 11161 catalog_manager.cc:1347] Generated new cluster ID: c16e7f0253d54dca96535883020b4414
I20260810 23:30:20.426271 11161 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:30:20.462301 11161 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:30:20.463714 11161 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:30:20.471546 11161 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2e035b11de047ec85beb4c68c77bec8: Generated new TSK 0
I20260810 23:30:20.472483 11161 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:30:20.478434 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.65:0
--local_ip_for_outbound_sockets=127.14.175.65
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:38331
--builtin_ntp_servers=127.14.175.84:35467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:30:20.868880 11179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:30:20.869223 11179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:30:20.869329 11179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:30:20.876701 11179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:30:20.876808 11179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:30:20.876922 11179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.65
I20260810 23:30:20.886333 11179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:35467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:38331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.65
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:30:20.887944 11179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:30:20.889869 11179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:30:20.903272 11187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:20.903218 11193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:20.903198 11188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:30:20.904263 11179 server_base.cc:1034] running on GCE node
I20260810 23:30:20.905017 11179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:30:20.906050 11179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:30:20.907269 11179 hybrid_clock.cc:648] HybridClock initialized: now 1786404620907209 us; error 55 us; skew 500 ppm
I20260810 23:30:20.907613 11179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:30:20.910034 11179 webserver.cc:460] Webserver started at http://127.14.175.65:44123/ using document root <none> and password file <none>
I20260810 23:30:20.910785 11179 fs_manager.cc:359] Metadata directory not provided
I20260810 23:30:20.910902 11179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:30:20.911227 11179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:30:20.914426 11179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-0/data/instance:
uuid: "3ec872414ab8401d938bcd16797ea5c1"
format_stamp: "Formatted at 2026-08-10 23:30:20 on dist-test-slave-rmhg"
server_key: "9773278d2b08a5b4f7a32abf537df10a"
server_key_iv: "9493740e490cd34d413e0de447ab0c21"
server_key_version: "encryptionkey@0"
I20260810 23:30:20.915308 11179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance:
uuid: "3ec872414ab8401d938bcd16797ea5c1"
format_stamp: "Formatted at 2026-08-10 23:30:20 on dist-test-slave-rmhg"
server_key: "9773278d2b08a5b4f7a32abf537df10a"
server_key_iv: "9493740e490cd34d413e0de447ab0c21"
server_key_version: "encryptionkey@0"
I20260810 23:30:20.921216 11179 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260810 23:30:20.924988 11204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:20.926504 11179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:30:20.926692 11179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-0/wal
uuid: "3ec872414ab8401d938bcd16797ea5c1"
format_stamp: "Formatted at 2026-08-10 23:30:20 on dist-test-slave-rmhg"
server_key: "9773278d2b08a5b4f7a32abf537df10a"
server_key_iv: "9493740e490cd34d413e0de447ab0c21"
server_key_version: "encryptionkey@0"
I20260810 23:30:20.926929 11179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:30:20.944226 11179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:30:20.945290 11179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:30:20.945618 11179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:30:20.946764 11179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:30:20.948397 11179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:30:20.948496 11179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:20.948609 11179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:30:20.948697 11179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:20.981058 11179 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.65:37063
I20260810 23:30:20.981142 11374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.65:37063 every 8 connection(s)
I20260810 23:30:20.982472 11179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
I20260810 23:30:20.986611 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 11179
I20260810 23:30:20.986812 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance
I20260810 23:30:20.987417 15037 external_mini_cluster.cc:1428] Setting key bd590da701228f9edd8900957957db20
I20260810 23:30:20.990636 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.66:0
--local_ip_for_outbound_sockets=127.14.175.66
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:38331
--builtin_ntp_servers=127.14.175.84:35467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:30:20.996941 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:20.985234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:52221 (local address 127.14.175.126:38331)
0810 23:30:20.985678 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:20.985683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:20.987227 (+ 1544us) server_negotiation.cc:407] Connection header received
0810 23:30:20.989523 (+ 2296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:20.989527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:20.989591 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:20.989697 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:20.991211 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:20.991929 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:20.994167 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:20.994569 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:20.994669 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:20.996210 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:20.996215 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:20.996219 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:20.996237 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:20.996280 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:20.996293 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:20.996359 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:20.996584 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 23:30:20.996670 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":210,"threads_started":1}
I20260810 23:30:20.998430 11384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:20.985866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:38331 (local address 127.14.175.65:52221)
0810 23:30:20.987040 (+ 1174us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:20.987091 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:20.989223 (+ 2132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:20.989998 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:20.990011 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:20.990143 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:30:20.991031 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:20.991058 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:20.992097 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:20.992102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:20.994031 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:20.994047 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:20.994804 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:20.994810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:20.995094 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:20.995841 (+ 747us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:30:20.995844 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:30:20.995901 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:30:20.996007 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:30:20.996379 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:30:20.996390 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:30:20.996397 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 23:30:20.996581 (+ 184us) client_negotiation.cc:240] Negotiation successful
0810 23:30:20.997040 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":331,"threads_started":1}
I20260810 23:30:21.000003 11378 heartbeater.cc:347] Connected to a master server at 127.14.175.126:38331
I20260810 23:30:21.000355 11378 heartbeater.cc:464] Registering TS with master...
I20260810 23:30:21.001369 11378 heartbeater.cc:511] Master 127.14.175.126:38331 requested a full tablet report, sending...
I20260810 23:30:21.003698 11087 ts_manager.cc:194] Registered new tserver with Master: 3ec872414ab8401d938bcd16797ea5c1 (127.14.175.65:37063)
I20260810 23:30:21.005869 11087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.175.65:52221
W20260810 23:30:21.352703 11388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:30:21.353024 11388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:30:21.353117 11388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:30:21.360949 11388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:30:21.361066 11388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:30:21.361236 11388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.66
I20260810 23:30:21.370827 11388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:35467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:38331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.66
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:30:21.372219 11388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:30:21.374233 11388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:30:21.388103 11396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:21.388141 11397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:21.388108 11400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:30:21.389060 11388 server_base.cc:1034] running on GCE node
I20260810 23:30:21.389770 11388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:30:21.390779 11388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:30:21.391989 11388 hybrid_clock.cc:648] HybridClock initialized: now 1786404621391926 us; error 56 us; skew 500 ppm
I20260810 23:30:21.392336 11388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:30:21.394765 11388 webserver.cc:460] Webserver started at http://127.14.175.66:36185/ using document root <none> and password file <none>
I20260810 23:30:21.395509 11388 fs_manager.cc:359] Metadata directory not provided
I20260810 23:30:21.395620 11388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:30:21.395933 11388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:30:21.399042 11388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-1/data/instance:
uuid: "a098fedc2a574bdb96aea039e8bb7841"
format_stamp: "Formatted at 2026-08-10 23:30:21 on dist-test-slave-rmhg"
server_key: "106fe910b7952acd8cbd1d4df8f0a9ff"
server_key_iv: "ae3b40d4961e02c3495d2d32c82d60f5"
server_key_version: "encryptionkey@0"
I20260810 23:30:21.399936 11388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance:
uuid: "a098fedc2a574bdb96aea039e8bb7841"
format_stamp: "Formatted at 2026-08-10 23:30:21 on dist-test-slave-rmhg"
server_key: "106fe910b7952acd8cbd1d4df8f0a9ff"
server_key_iv: "ae3b40d4961e02c3495d2d32c82d60f5"
server_key_version: "encryptionkey@0"
I20260810 23:30:21.405815 11388 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260810 23:30:21.409817 11409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:21.411363 11388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 23:30:21.411543 11388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-1/wal
uuid: "a098fedc2a574bdb96aea039e8bb7841"
format_stamp: "Formatted at 2026-08-10 23:30:21 on dist-test-slave-rmhg"
server_key: "106fe910b7952acd8cbd1d4df8f0a9ff"
server_key_iv: "ae3b40d4961e02c3495d2d32c82d60f5"
server_key_version: "encryptionkey@0"
I20260810 23:30:21.411784 11388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:30:21.443498 11388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:30:21.444411 11388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:30:21.444722 11388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:30:21.445833 11388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:30:21.447397 11388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:30:21.447491 11388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:21.447611 11388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:30:21.447681 11388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:21.478955 11388 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.66:35465
I20260810 23:30:21.479028 11574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.66:35465 every 8 connection(s)
I20260810 23:30:21.480302 11388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
I20260810 23:30:21.485991 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 11388
I20260810 23:30:21.486166 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance
I20260810 23:30:21.486609 15037 external_mini_cluster.cc:1428] Setting key 3a45c33a9dbf00e7a6973767d2da83d5
I20260810 23:30:21.489918 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.67:0
--local_ip_for_outbound_sockets=127.14.175.67
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:38331
--builtin_ntp_servers=127.14.175.84:35467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:30:21.494537 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:21.482895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:58967 (local address 127.14.175.126:38331)
0810 23:30:21.483124 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:21.483139 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:21.485085 (+ 1946us) server_negotiation.cc:407] Connection header received
0810 23:30:21.487070 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:21.487076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:21.487171 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:21.487264 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:21.488784 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:21.489879 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:21.491602 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:21.492246 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:21.492369 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:21.493776 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:21.493781 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:21.493785 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:21.493808 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:21.493845 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:21.493859 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:21.493943 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:21.494137 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 23:30:21.494239 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 23:30:21.495581 11577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:21.483449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:38331 (local address 127.14.175.66:58967)
0810 23:30:21.484434 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:21.484949 (+ 515us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:21.486782 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:21.487565 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:21.487575 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:21.487702 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:30:21.488575 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:21.488607 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:21.490071 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:21.490077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:21.491439 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:21.491453 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:21.492472 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:21.492477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:21.492707 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:21.493422 (+ 715us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:30:21.493425 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:30:21.493479 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:30:21.493570 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:30:21.493949 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:30:21.493956 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:30:21.493961 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:30:21.494102 (+ 141us) client_negotiation.cc:240] Negotiation successful
0810 23:30:21.494460 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":310,"threads_started":1}
I20260810 23:30:21.497046 11575 heartbeater.cc:347] Connected to a master server at 127.14.175.126:38331
I20260810 23:30:21.497412 11575 heartbeater.cc:464] Registering TS with master...
I20260810 23:30:21.498339 11575 heartbeater.cc:511] Master 127.14.175.126:38331 requested a full tablet report, sending...
I20260810 23:30:21.499926 11087 ts_manager.cc:194] Registered new tserver with Master: a098fedc2a574bdb96aea039e8bb7841 (127.14.175.66:35465)
I20260810 23:30:21.500754 11087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.175.66:58967
W20260810 23:30:21.843976 11578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:30:21.844317 11578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:30:21.844439 11578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:30:21.852016 11578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:30:21.852145 11578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:30:21.852298 11578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.67
I20260810 23:30:21.861814 11578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:35467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:38331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.67
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:30:21.863221 11578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:30:21.865195 11578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:30:21.878749 11584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:21.878733 11585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:21.878734 11588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:30:21.879637 11578 server_base.cc:1034] running on GCE node
I20260810 23:30:21.880304 11578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:30:21.881347 11578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:30:21.882565 11578 hybrid_clock.cc:648] HybridClock initialized: now 1786404621882512 us; error 45 us; skew 500 ppm
I20260810 23:30:21.882902 11578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:30:21.885337 11578 webserver.cc:460] Webserver started at http://127.14.175.67:39119/ using document root <none> and password file <none>
I20260810 23:30:21.886081 11578 fs_manager.cc:359] Metadata directory not provided
I20260810 23:30:21.886193 11578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:30:21.886504 11578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:30:21.889482 11578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-2/data/instance:
uuid: "8e7bdd40f5604165a932f5050ed70032"
format_stamp: "Formatted at 2026-08-10 23:30:21 on dist-test-slave-rmhg"
server_key: "765a252d780c26ab97394cab39ccf95f"
server_key_iv: "e078fe67f189f836b87ea1f202ccd82c"
server_key_version: "encryptionkey@0"
I20260810 23:30:21.890287 11578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance:
uuid: "8e7bdd40f5604165a932f5050ed70032"
format_stamp: "Formatted at 2026-08-10 23:30:21 on dist-test-slave-rmhg"
server_key: "765a252d780c26ab97394cab39ccf95f"
server_key_iv: "e078fe67f189f836b87ea1f202ccd82c"
server_key_version: "encryptionkey@0"
I20260810 23:30:21.895481 11578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:30:21.899093 11600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:21.900449 11578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 23:30:21.900610 11578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-2/wal
uuid: "8e7bdd40f5604165a932f5050ed70032"
format_stamp: "Formatted at 2026-08-10 23:30:21 on dist-test-slave-rmhg"
server_key: "765a252d780c26ab97394cab39ccf95f"
server_key_iv: "e078fe67f189f836b87ea1f202ccd82c"
server_key_version: "encryptionkey@0"
I20260810 23:30:21.900818 11578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:30:21.917462 11578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:30:21.918322 11578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:30:21.918612 11578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:30:21.921764 11578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:30:21.923409 11578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:30:21.923518 11578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:21.923632 11578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:30:21.923712 11578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:21.954276 11578 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.67:40777
I20260810 23:30:21.954319 11771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.67:40777 every 8 connection(s)
I20260810 23:30:21.955585 11578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
I20260810 23:30:21.962390 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 11578
I20260810 23:30:21.962577 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance
I20260810 23:30:21.963137 15037 external_mini_cluster.cc:1428] Setting key 5c700f0752260c81bd13668113e6d375
I20260810 23:30:21.968747 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:21.958179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:38931 (local address 127.14.175.126:38331)
0810 23:30:21.958398 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:21.958404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:21.959897 (+ 1493us) server_negotiation.cc:407] Connection header received
0810 23:30:21.961678 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:21.961683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:21.961764 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:21.961854 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:21.962979 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:21.963980 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:21.966169 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:21.966657 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:21.966757 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:21.968075 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:21.968080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:21.968083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:21.968101 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:21.968129 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:21.968141 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:21.968205 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:21.968387 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 23:30:21.968470 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 23:30:21.969801 11775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:21.958781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:38331 (local address 127.14.175.67:38931)
0810 23:30:21.959735 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:21.959774 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:21.961429 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:21.962044 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:21.962052 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:21.962140 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:30:21.962789 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:21.962812 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:21.964159 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:21.964165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:21.966008 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:21.966025 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:21.966865 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:21.966871 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:21.967084 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:21.967776 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:30:21.967779 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:30:21.967832 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:30:21.967918 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:30:21.968216 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:30:21.968224 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:30:21.968230 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 23:30:21.968369 (+ 139us) client_negotiation.cc:240] Negotiation successful
0810 23:30:21.968703 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":307,"threads_started":1}
I20260810 23:30:21.971171 11772 heartbeater.cc:347] Connected to a master server at 127.14.175.126:38331
I20260810 23:30:21.971472 11772 heartbeater.cc:464] Registering TS with master...
I20260810 23:30:21.972292 11772 heartbeater.cc:511] Master 127.14.175.126:38331 requested a full tablet report, sending...
I20260810 23:30:21.973742 11087 ts_manager.cc:194] Registered new tserver with Master: 8e7bdd40f5604165a932f5050ed70032 (127.14.175.67:40777)
I20260810 23:30:21.974529 11087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.175.67:38931
I20260810 23:30:21.978780 15037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:30:21.990716 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:21.983563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49146 (local address 127.14.175.126:38331)
0810 23:30:21.983822 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:21.983829 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:21.984224 (+ 395us) server_negotiation.cc:407] Connection header received
0810 23:30:21.984373 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:21.984378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:21.984479 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:21.984613 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:21.985659 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:21.986722 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:21.988438 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:21.988851 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:21.988958 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:21.990119 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:21.990124 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:21.990128 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:21.990149 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:21.990185 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:21.990197 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:21.990266 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:21.990379 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 23:30:21.990462 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 23:30:21.994722 11087 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:30:22.001775 11087 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:30:22.009454 11378 heartbeater.cc:503] Master 127.14.175.126:38331 was elected leader, sending a full tablet report...
I20260810 23:30:22.029377 11797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:22.015199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43250 (local address 127.14.175.67:40777)
0810 23:30:22.022123 (+ 6924us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:22.022143 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:22.022180 (+ 37us) server_negotiation.cc:407] Connection header received
0810 23:30:22.022286 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:22.022293 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:22.022526 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:22.022721 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:30:22.023292 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:22.024820 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:22.026782 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:22.028148 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:22.028439 (+ 291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:22.028697 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:22.028771 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 23:30:22.028945 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6741,"thread_start_us":6585,"threads_started":1}
I20260810 23:30:22.029393 11793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:22.015079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:40777 (local address 127.0.0.1:43250)
0810 23:30:22.015943 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:22.016072 (+ 129us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:22.016266 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:22.022736 (+ 6470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:22.022741 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:22.022767 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:30:22.023123 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:22.023134 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:22.025027 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:22.025032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:22.026570 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:22.026585 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:22.028366 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:22.028372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:22.028603 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:22.028648 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 23:30:22.028710 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 23:30:22.028917 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":85,"thread_start_us":433,"threads_started":1}
I20260810 23:30:22.029403 11799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:22.015128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:37063 (local address 127.0.0.1:38714)
0810 23:30:22.017583 (+ 2455us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:22.017598 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:22.017695 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:22.020956 (+ 3261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:22.020965 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:22.020997 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:30:22.021492 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:22.021509 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:22.023552 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:22.023558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:22.026281 (+ 2723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:22.026299 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:22.028167 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:22.028173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:22.028449 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:22.028530 (+ 81us) client_negotiation.cc:769] Sending connection context
0810 23:30:22.028655 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 23:30:22.028917 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2342,"mutex_wait_us":80,"thread_start_us":4904,"threads_started":1}
I20260810 23:30:22.029645 11802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:22.016879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51596 (local address 127.14.175.66:35465)
0810 23:30:22.017451 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:22.017460 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:22.020592 (+ 3132us) server_negotiation.cc:407] Connection header received
0810 23:30:22.020815 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:22.020823 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:22.021130 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:22.021376 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:30:22.022090 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:22.023372 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:22.026550 (+ 3178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:22.027991 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:22.028129 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:22.028395 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:22.029110 (+ 715us) server_negotiation.cc:299] Negotiation successful
0810 23:30:22.029303 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":320,"threads_started":1}
I20260810 23:30:22.030206 11792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:22.016355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:35465 (local address 127.0.0.1:51596)
0810 23:30:22.020314 (+ 3959us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:22.020331 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:22.020459 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:22.021463 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:22.021469 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:22.021499 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:30:22.021902 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:22.021912 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:22.023555 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:22.023560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:22.026286 (+ 2726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:22.026299 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:22.028525 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:22.028532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:22.028757 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:22.028789 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 23:30:22.028870 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 23:30:22.028991 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3840,"mutex_wait_us":223,"thread_start_us":330,"threads_started":1}
I20260810 23:30:22.030786 11798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:22.015756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38714 (local address 127.14.175.65:37063)
0810 23:30:22.020111 (+ 4355us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:22.020121 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:22.020169 (+ 48us) server_negotiation.cc:407] Connection header received
0810 23:30:22.020288 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:22.020298 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:22.020602 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:22.021681 (+ 1079us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:30:22.021972 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:22.023386 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:22.026529 (+ 3143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:22.027904 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:22.029824 (+ 1920us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:22.030102 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:22.030188 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 23:30:22.030421 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4143,"thread_start_us":4004,"threads_started":1}
I20260810 23:30:22.034109 11479 tablet_service.cc:1467] Processing CreateTablet for tablet 35832babe63d4dc7bedce1994c9c3ed4 (DEFAULT_TABLE table=test-table [id=48efdd6ff2ff41de8f6cd58ec75d49c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:30:22.034143 11265 tablet_service.cc:1467] Processing CreateTablet for tablet 35832babe63d4dc7bedce1994c9c3ed4 (DEFAULT_TABLE table=test-table [id=48efdd6ff2ff41de8f6cd58ec75d49c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:30:22.034143 11681 tablet_service.cc:1467] Processing CreateTablet for tablet 35832babe63d4dc7bedce1994c9c3ed4 (DEFAULT_TABLE table=test-table [id=48efdd6ff2ff41de8f6cd58ec75d49c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:30:22.035518 11479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35832babe63d4dc7bedce1994c9c3ed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:30:22.035588 11681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35832babe63d4dc7bedce1994c9c3ed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:30:22.035602 11265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35832babe63d4dc7bedce1994c9c3ed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:30:22.048048 11804 tablet_bootstrap.cc:492] T 35832babe63d4dc7bedce1994c9c3ed4 P a098fedc2a574bdb96aea039e8bb7841: Bootstrap starting.
I20260810 23:30:22.050346 11805 tablet_bootstrap.cc:492] T 35832babe63d4dc7bedce1994c9c3ed4 P 8e7bdd40f5604165a932f5050ed70032: Bootstrap starting.
I20260810 23:30:22.052196 11804 tablet_bootstrap.cc:654] T 35832babe63d4dc7bedce1994c9c3ed4 P a098fedc2a574bdb96aea039e8bb7841: Neither blocks nor log segments found. Creating new log.
I20260810 23:30:22.053851 11804 log.cc:824] T 35832babe63d4dc7bedce1994c9c3ed4 P a098fedc2a574bdb96aea039e8bb7841: Log is configured to *not* fsync() on all Append() calls
I20260810 23:30:22.054572 11805 tablet_bootstrap.cc:654] T 35832babe63d4dc7bedce1994c9c3ed4 P 8e7bdd40f5604165a932f5050ed70032: Neither blocks nor log segments found. Creating new log.
I20260810 23:30:22.056054 11805 log.cc:824] T 35832babe63d4dc7bedce1994c9c3ed4 P 8e7bdd40f5604165a932f5050ed70032: Log is configured to *not* fsync() on all Append() calls
I20260810 23:30:22.058257 11804 tablet_bootstrap.cc:492] T 35832babe63d4dc7bedce1994c9c3ed4 P a098fedc2a574bdb96aea039e8bb7841: No bootstrap required, opened a new log
I20260810 23:30:22.058598 11804 ts_tablet_manager.cc:1397] T 35832babe63d4dc7bedce1994c9c3ed4 P a098fedc2a574bdb96aea039e8bb7841: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260810 23:30:22.059422 11805 tablet_bootstrap.cc:492] T 35832babe63d4dc7bedce1994c9c3ed4 P 8e7bdd40f5604165a932f5050ed70032: No bootstrap required, opened a new log
I20260810 23:30:22.059708 11805 ts_tablet_manager.cc:1397] T 35832babe63d4dc7bedce1994c9c3ed4 P 8e7bdd40f5604165a932f5050ed70032: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260810 23:30:22.061616 11806 tablet_bootstrap.cc:492] T 35832babe63d4dc7bedce1994c9c3ed4 P 3ec872414ab8401d938bcd16797ea5c1: Bootstrap starting.
I20260810 23:30:22.064684 11806 tablet_bootstrap.cc:654] T 35832babe63d4dc7bedce1994c9c3ed4 P 3ec872414ab8401d938bcd16797ea5c1: Neither blocks nor log segments found. Creating new log.
I20260810 23:30:22.065784 11806 log.cc:824] T 35832babe63d4dc7bedce1994c9c3ed4 P 3ec872414ab8401d938bcd16797ea5c1: Log is configured to *not* fsync() on all Append() calls
I20260810 23:30:22.068559 11806 tablet_bootstrap.cc:492] T 35832babe63d4dc7bedce1994c9c3ed4 P 3ec872414ab8401d938bcd16797ea5c1: No bootstrap required, opened a new log
I20260810 23:30:22.068830 11806 ts_tablet_manager.cc:1397] T 35832babe63d4dc7bedce1994c9c3ed4 P 3ec872414ab8401d938bcd16797ea5c1: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20260810 23:30:22.069489 11804 raft_consensus.cc:348] T 35832babe63d4dc7bedce1994c9c3ed4 P a098fedc2a574bdb96aea039e8bb7841 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e7bdd40f5604165a932f5050ed70032" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40777 } } peers { permanent_uuid: "3ec872414ab8401d938bcd16797ea5c1" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 37063 } } peers { permanent_uuid: "a098fedc2a574bdb96aea039e8bb7841" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 35465 } }
I20260810 23:30:22.070001 11804 raft_consensus.cc:374] T 35832babe63d4dc7bedce1994c9c3ed4 P a098fedc2a574bdb96aea039e8bb7841 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:30:22.070117 11804 raft_consensus.cc:729] T 35832babe63d4dc7bedce1994c9c3ed4 P a098fedc2a574bdb96aea039e8bb7841 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a098fedc2a574bdb96aea039e8bb7841, State: Initialized, Role: FOLLOWER
I20260810 23:30:22.070210 11805 raft_consensus.cc:348] T 35832babe63d4dc7bedce1994c9c3ed4 P 8e7bdd40f5604165a932f5050ed70032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e7bdd40f5604165a932f5050ed70032" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40777 } } peers { permanent_uuid: "3ec872414ab8401d938bcd16797ea5c1" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 37063 } } peers { permanent_uuid: "a098fedc2a574bdb96aea039e8bb7841" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 35465 } }
I20260810 23:30:22.070636 11805 raft_consensus.cc:374] T 35832babe63d4dc7bedce1994c9c3ed4 P 8e7bdd40f5604165a932f5050ed70032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:30:22.070737 11805 raft_consensus.cc:729] T 35832babe63d4dc7bedce1994c9c3ed4 P 8e7bdd40f5604165a932f5050ed70032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7bdd40f5604165a932f5050ed70032, State: Initialized, Role: FOLLOWER
I20260810 23:30:22.070811 11804 consensus_queue.cc:260] T 35832babe63d4dc7bedce1994c9c3ed4 P a098fedc2a574bdb96aea039e8bb7841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e7bdd40f5604165a932f5050ed70032" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40777 } } peers { permanent_uuid: "3ec872414ab8401d938bcd16797ea5c1" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 37063 } } peers { permanent_uuid: "a098fedc2a574bdb96aea039e8bb7841" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 35465 } }
I20260810 23:30:22.071435 11805 consensus_queue.cc:260] T 35832babe63d4dc7bedce1994c9c3ed4 P 8e7bdd40f5604165a932f5050ed70032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e7bdd40f5604165a932f5050ed70032" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40777 } } peers { permanent_uuid: "3ec872414ab8401d938bcd16797ea5c1" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 37063 } } peers { permanent_uuid: "a098fedc2a574bdb96aea039e8bb7841" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 35465 } }
I20260810 23:30:22.076764 11772 heartbeater.cc:503] Master 127.14.175.126:38331 was elected leader, sending a full tablet report...
I20260810 23:30:22.076983 11806 raft_consensus.cc:348] T 35832babe63d4dc7bedce1994c9c3ed4 P 3ec872414ab8401d938bcd16797ea5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e7bdd40f5604165a932f5050ed70032" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40777 } } peers { permanent_uuid: "3ec872414ab8401d938bcd16797ea5c1" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 37063 } } peers { permanent_uuid: "a098fedc2a574bdb96aea039e8bb7841" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 35465 } }
I20260810 23:30:22.077486 11806 raft_consensus.cc:374] T 35832babe63d4dc7bedce1994c9c3ed4 P 3ec872414ab8401d938bcd16797ea5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:30:22.077582 11806 raft_consensus.cc:729] T 35832babe63d4dc7bedce1994c9c3ed4 P 3ec872414ab8401d938bcd16797ea5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ec872414ab8401d938bcd16797ea5c1, State: Initialized, Role: FOLLOWER
I20260810 23:30:22.077999 11805 ts_tablet_manager.cc:1428] T 35832babe63d4dc7bedce1994c9c3ed4 P 8e7bdd40f5604165a932f5050ed70032: Time spent starting tablet: real 0.018s user 0.018s sys 0.000s
I20260810 23:30:22.078339 11806 consensus_queue.cc:260] T 35832babe63d4dc7bedce1994c9c3ed4 P 3ec872414ab8401d938bcd16797ea5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e7bdd40f5604165a932f5050ed70032" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40777 } } peers { permanent_uuid: "3ec872414ab8401d938bcd16797ea5c1" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 37063 } } peers { permanent_uuid: "a098fedc2a574bdb96aea039e8bb7841" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 35465 } }
I20260810 23:30:22.079002 11575 heartbeater.cc:503] Master 127.14.175.126:38331 was elected leader, sending a full tablet report...
I20260810 23:30:22.081831 11804 ts_tablet_manager.cc:1428] T 35832babe63d4dc7bedce1994c9c3ed4 P a098fedc2a574bdb96aea039e8bb7841: Time spent starting tablet: real 0.023s user 0.018s sys 0.004s
I20260810 23:30:22.082973 11806 ts_tablet_manager.cc:1428] T 35832babe63d4dc7bedce1994c9c3ed4 P 3ec872414ab8401d938bcd16797ea5c1: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260810 23:30:22.128458 11814 raft_consensus.cc:482] T 35832babe63d4dc7bedce1994c9c3ed4 P a098fedc2a574bdb96aea039e8bb7841 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:30:22.128880 11814 raft_consensus.cc:504] T 35832babe63d4dc7bedce1994c9c3ed4 P a098fedc2a574bdb96aea039e8bb7841 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e7bdd40f5604165a932f5050ed70032" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40777 } } peers { permanent_uuid: "3ec872414ab8401d938bcd16797ea5c1" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 37063 } } peers { permanent_uuid: "a098fedc2a574bdb96aea039e8bb7841" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 35465 } }
I20260810 23:30:22.130810 11814 leader_election.cc:288] T 35832babe63d4dc7bedce1994c9c3ed4 P a098fedc2a574bdb96aea039e8bb7841 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e7bdd40f5604165a932f5050ed70032 (127.14.175.67:40777), 3ec872414ab8401d938bcd16797ea5c1 (127.14.175.65:37063)
I20260810 23:30:22.137849 11798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:22.131790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:60271 (local address 127.14.175.65:37063)
0810 23:30:22.131984 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:22.131990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:22.132012 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:30:22.132299 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:22.132304 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:22.132381 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:22.132434 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:30:22.133214 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:22.134072 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:22.136076 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:22.136881 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:22.137060 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:22.137173 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:22.137476 (+ 303us) server_negotiation.cc:299] Negotiation successful
0810 23:30:22.137610 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 23:30:22.137959 11818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:22.131159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:37063 (local address 127.14.175.66:60271)
0810 23:30:22.131890 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:22.131912 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:22.132090 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:22.132608 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:22.132611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:22.132634 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:30:22.132983 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:22.133001 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:22.134270 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:22.134274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:22.135862 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:22.135877 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:22.137127 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:22.137134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:22.137318 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:22.137343 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 23:30:22.137425 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 23:30:22.137562 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"mutex_wait_us":4,"thread_start_us":838,"threads_started":1}
I20260810 23:30:22.138933 11309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35832babe63d4dc7bedce1994c9c3ed4" candidate_uuid: "a098fedc2a574bdb96aea039e8bb7841" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ec872414ab8401d938bcd16797ea5c1" is_pre_election: true
I20260810 23:30:22.139104 11797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:22.131561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:52751 (local address 127.14.175.67:40777)
0810 23:30:22.131811 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:22.131819 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:22.133739 (+ 1920us) server_negotiation.cc:407] Connection header received
0810 23:30:22.133883 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:22.133888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:22.133949 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:22.134052 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:30:22.134672 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:22.135421 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:22.137266 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:22.138153 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:22.138308 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:22.138375 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:22.138745 (+ 370us) server_negotiation.cc:299] Negotiation successful
0810 23:30:22.138876 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260810 23:30:22.139146 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:22.131159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:40777 (local address 127.14.175.66:52751)
0810 23:30:22.133620 (+ 2461us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:22.133637 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:22.133761 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:22.134066 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:22.134071 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:22.134094 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:30:22.134506 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:22.134518 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:22.135666 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:22.135671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:22.137036 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:22.137053 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:22.138444 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:22.138449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:22.138580 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:22.138603 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 23:30:22.138683 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 23:30:22.138811 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2349,"mutex_wait_us":72,"thread_start_us":2086,"threads_started":1}
I20260810 23:30:22.139544 11309 raft_consensus.cc:2454] T 35832babe63d4dc7bedce1994c9c3ed4 P 3ec872414ab8401d938bcd16797ea5c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a098fedc2a574bdb96aea039e8bb7841 in term 0.
I20260810 23:30:22.140316 11410 leader_election.cc:302] T 35832babe63d4dc7bedce1994c9c3ed4 P a098fedc2a574bdb96aea039e8bb7841 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ec872414ab8401d938bcd16797ea5c1, a098fedc2a574bdb96aea039e8bb7841; no voters:
I20260810 23:30:22.140316 11715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35832babe63d4dc7bedce1994c9c3ed4" candidate_uuid: "a098fedc2a574bdb96aea039e8bb7841" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7bdd40f5604165a932f5050ed70032" is_pre_election: true
I20260810 23:30:22.140825 11814 raft_consensus.cc:2781] T 35832babe63d4dc7bedce1994c9c3ed4 P a098fedc2a574bdb96aea039e8bb7841 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:30:22.140982 11814 raft_consensus.cc:482] T 35832babe63d4dc7bedce1994c9c3ed4 P a098fedc2a574bdb96aea039e8bb7841 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:30:22.140914 11715 raft_consensus.cc:2454] T 35832babe63d4dc7bedce1994c9c3ed4 P 8e7bdd40f5604165a932f5050ed70032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a098fedc2a574bdb96aea039e8bb7841 in term 0.
I20260810 23:30:22.141093 11814 raft_consensus.cc:3037] T 35832babe63d4dc7bedce1994c9c3ed4 P a098fedc2a574bdb96aea039e8bb7841 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:30:22.143567 11814 raft_consensus.cc:504] T 35832babe63d4dc7bedce1994c9c3ed4 P a098fedc2a574bdb96aea039e8bb7841 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e7bdd40f5604165a932f5050ed70032" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40777 } } peers { permanent_uuid: "3ec872414ab8401d938bcd16797ea5c1" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 37063 } } peers { permanent_uuid: "a098fedc2a574bdb96aea039e8bb7841" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 35465 } }
I20260810 23:30:22.144205 11814 leader_election.cc:288] T 35832babe63d4dc7bedce1994c9c3ed4 P a098fedc2a574bdb96aea039e8bb7841 [CANDIDATE]: Term 1 election: Requested vote from peers 8e7bdd40f5604165a932f5050ed70032 (127.14.175.67:40777), 3ec872414ab8401d938bcd16797ea5c1 (127.14.175.65:37063)
I20260810 23:30:22.144840 11715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35832babe63d4dc7bedce1994c9c3ed4" candidate_uuid: "a098fedc2a574bdb96aea039e8bb7841" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7bdd40f5604165a932f5050ed70032"
I20260810 23:30:22.144851 11309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35832babe63d4dc7bedce1994c9c3ed4" candidate_uuid: "a098fedc2a574bdb96aea039e8bb7841" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ec872414ab8401d938bcd16797ea5c1"
I20260810 23:30:22.145058 11715 raft_consensus.cc:3037] T 35832babe63d4dc7bedce1994c9c3ed4 P 8e7bdd40f5604165a932f5050ed70032 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:30:22.145115 11309 raft_consensus.cc:3037] T 35832babe63d4dc7bedce1994c9c3ed4 P 3ec872414ab8401d938bcd16797ea5c1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:30:22.147797 11715 raft_consensus.cc:2454] T 35832babe63d4dc7bedce1994c9c3ed4 P 8e7bdd40f5604165a932f5050ed70032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a098fedc2a574bdb96aea039e8bb7841 in term 1.
I20260810 23:30:22.147871 11309 raft_consensus.cc:2454] T 35832babe63d4dc7bedce1994c9c3ed4 P 3ec872414ab8401d938bcd16797ea5c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a098fedc2a574bdb96aea039e8bb7841 in term 1.
I20260810 23:30:22.148304 11414 leader_election.cc:302] T 35832babe63d4dc7bedce1994c9c3ed4 P a098fedc2a574bdb96aea039e8bb7841 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7bdd40f5604165a932f5050ed70032, a098fedc2a574bdb96aea039e8bb7841; no voters:
I20260810 23:30:22.148700 11814 raft_consensus.cc:2781] T 35832babe63d4dc7bedce1994c9c3ed4 P a098fedc2a574bdb96aea039e8bb7841 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:30:22.149183 11814 raft_consensus.cc:686] T 35832babe63d4dc7bedce1994c9c3ed4 P a098fedc2a574bdb96aea039e8bb7841 [term 1 LEADER]: Becoming Leader. State: Replica: a098fedc2a574bdb96aea039e8bb7841, State: Running, Role: LEADER
I20260810 23:30:22.149677 11814 consensus_queue.cc:237] T 35832babe63d4dc7bedce1994c9c3ed4 P a098fedc2a574bdb96aea039e8bb7841 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e7bdd40f5604165a932f5050ed70032" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40777 } } peers { permanent_uuid: "3ec872414ab8401d938bcd16797ea5c1" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 37063 } } peers { permanent_uuid: "a098fedc2a574bdb96aea039e8bb7841" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 35465 } }
I20260810 23:30:22.154340 11084 catalog_manager.cc:5495] T 35832babe63d4dc7bedce1994c9c3ed4 P a098fedc2a574bdb96aea039e8bb7841 reported cstate change: term changed from 0 to 1, leader changed from <none> to a098fedc2a574bdb96aea039e8bb7841 (127.14.175.66). New cstate: current_term: 1 leader_uuid: "a098fedc2a574bdb96aea039e8bb7841" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e7bdd40f5604165a932f5050ed70032" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ec872414ab8401d938bcd16797ea5c1" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a098fedc2a574bdb96aea039e8bb7841" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 35465 } health_report { overall_health: HEALTHY } } }
I20260810 23:30:22.189976 11802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:22.184287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51610 (local address 127.14.175.66:35465)
0810 23:30:22.184527 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:22.184532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:22.184553 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:30:22.184686 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:22.184690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:22.184786 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:22.184926 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:30:22.185362 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:22.186071 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:22.187547 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:22.187984 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:22.188103 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:22.188659 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:30:22.189228 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:30:22.189379 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:22.189490 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 23:30:22.189575 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 23:30:22.198541 11715 raft_consensus.cc:1261] T 35832babe63d4dc7bedce1994c9c3ed4 P 8e7bdd40f5604165a932f5050ed70032 [term 1 FOLLOWER]: Refusing update from remote peer a098fedc2a574bdb96aea039e8bb7841: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:30:22.198549 11309 raft_consensus.cc:1261] T 35832babe63d4dc7bedce1994c9c3ed4 P 3ec872414ab8401d938bcd16797ea5c1 [term 1 FOLLOWER]: Refusing update from remote peer a098fedc2a574bdb96aea039e8bb7841: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:30:22.199553 11823 consensus_queue.cc:1035] T 35832babe63d4dc7bedce1994c9c3ed4 P a098fedc2a574bdb96aea039e8bb7841 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ec872414ab8401d938bcd16797ea5c1" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:30:22.199891 11814 consensus_queue.cc:1035] T 35832babe63d4dc7bedce1994c9c3ed4 P a098fedc2a574bdb96aea039e8bb7841 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7bdd40f5604165a932f5050ed70032" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 40777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 23:30:22.207147 11774 tablet.cc:2369] T 35832babe63d4dc7bedce1994c9c3ed4 P 8e7bdd40f5604165a932f5050ed70032: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:30:22.229907 11087 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:30:22.230195 11087 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:30:22.234308 11087 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2e035b11de047ec85beb4c68c77bec8: Sending DeleteTablet for 3 replicas of tablet 35832babe63d4dc7bedce1994c9c3ed4
I20260810 23:30:22.236109 11681 tablet_service.cc:1514] Processing DeleteTablet for tablet 35832babe63d4dc7bedce1994c9c3ed4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:30:22 UTC) from {username='slave'} at 127.0.0.1:43250
I20260810 23:30:22.236116 11265 tablet_service.cc:1514] Processing DeleteTablet for tablet 35832babe63d4dc7bedce1994c9c3ed4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:30:22 UTC) from {username='slave'} at 127.0.0.1:38714
I20260810 23:30:22.237092 11479 tablet_service.cc:1514] Processing DeleteTablet for tablet 35832babe63d4dc7bedce1994c9c3ed4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:30:22 UTC) from {username='slave'} at 127.0.0.1:51596
I20260810 23:30:22.238646 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 11179
I20260810 23:30:22.246904 11844 tablet_replica.cc:330] T 35832babe63d4dc7bedce1994c9c3ed4 P a098fedc2a574bdb96aea039e8bb7841: stopping tablet replica
I20260810 23:30:22.247502 11844 raft_consensus.cc:2229] T 35832babe63d4dc7bedce1994c9c3ed4 P a098fedc2a574bdb96aea039e8bb7841 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:30:22.247906 11841 tablet_replica.cc:330] T 35832babe63d4dc7bedce1994c9c3ed4 P 8e7bdd40f5604165a932f5050ed70032: stopping tablet replica
I20260810 23:30:22.247612 11843 tablet_replica.cc:330] T 35832babe63d4dc7bedce1994c9c3ed4 P 3ec872414ab8401d938bcd16797ea5c1: stopping tablet replica
I20260810 23:30:22.248296 11844 raft_consensus.cc:2258] T 35832babe63d4dc7bedce1994c9c3ed4 P a098fedc2a574bdb96aea039e8bb7841 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:30:22.248833 11841 raft_consensus.cc:2229] T 35832babe63d4dc7bedce1994c9c3ed4 P 8e7bdd40f5604165a932f5050ed70032 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:30:22.249143 11843 raft_consensus.cc:2229] T 35832babe63d4dc7bedce1994c9c3ed4 P 3ec872414ab8401d938bcd16797ea5c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:30:22.249768 11841 raft_consensus.cc:2258] T 35832babe63d4dc7bedce1994c9c3ed4 P 8e7bdd40f5604165a932f5050ed70032 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:30:22.250298 11843 raft_consensus.cc:2258] T 35832babe63d4dc7bedce1994c9c3ed4 P 3ec872414ab8401d938bcd16797ea5c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:30:22.250856 11844 ts_tablet_manager.cc:1905] T 35832babe63d4dc7bedce1994c9c3ed4 P a098fedc2a574bdb96aea039e8bb7841: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:30:22.252888 11843 ts_tablet_manager.cc:1905] T 35832babe63d4dc7bedce1994c9c3ed4 P 3ec872414ab8401d938bcd16797ea5c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:30:22.252884 11841 ts_tablet_manager.cc:1905] T 35832babe63d4dc7bedce1994c9c3ed4 P 8e7bdd40f5604165a932f5050ed70032: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:30:22.255002 11798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:22.240433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38726 (local address 127.14.175.65:37063)
0810 23:30:22.240654 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:22.240661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:22.244795 (+ 4134us) server_negotiation.cc:407] Connection header received
0810 23:30:22.245955 (+ 1160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:22.245964 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:22.246110 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:22.246194 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:22.247447 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:22.248599 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:22.250585 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:22.251275 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:22.251377 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:22.253178 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:22.253192 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:22.253210 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:22.254002 (+ 792us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:22.254068 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:22.254094 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:22.254201 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:22.254469 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 23:30:22.254604 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 23:30:22.256214 11368 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:30:22.257793 11844 ts_tablet_manager.cc:1918] T 35832babe63d4dc7bedce1994c9c3ed4 P a098fedc2a574bdb96aea039e8bb7841: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:30:22.258004 11844 log.cc:1196] T 35832babe63d4dc7bedce1994c9c3ed4 P a098fedc2a574bdb96aea039e8bb7841: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-1/wal/wals/35832babe63d4dc7bedce1994c9c3ed4
I20260810 23:30:22.258764 11844 ts_tablet_manager.cc:1939] T 35832babe63d4dc7bedce1994c9c3ed4 P a098fedc2a574bdb96aea039e8bb7841: Deleting consensus metadata
I20260810 23:30:22.260386 11841 ts_tablet_manager.cc:1918] T 35832babe63d4dc7bedce1994c9c3ed4 P 8e7bdd40f5604165a932f5050ed70032: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:30:22.260564 11841 log.cc:1196] T 35832babe63d4dc7bedce1994c9c3ed4 P 8e7bdd40f5604165a932f5050ed70032: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404299514076-15037-0/minicluster-data/ts-2/wal/wals/35832babe63d4dc7bedce1994c9c3ed4
I20260810 23:30:22.260771 11067 catalog_manager.cc:4841] TS a098fedc2a574bdb96aea039e8bb7841 (127.14.175.66:35465): tablet 35832babe63d4dc7bedce1994c9c3ed4 (table test-table [id=48efdd6ff2ff41de8f6cd58ec75d49c9]) successfully deleted
I20260810 23:30:22.261121 11841 ts_tablet_manager.cc:1939] T 35832babe63d4dc7bedce1994c9c3ed4 P 8e7bdd40f5604165a932f5050ed70032: Deleting consensus metadata
I20260810 23:30:22.263162 11067 catalog_manager.cc:4841] TS 8e7bdd40f5604165a932f5050ed70032 (127.14.175.67:40777): tablet 35832babe63d4dc7bedce1994c9c3ed4 (table test-table [id=48efdd6ff2ff41de8f6cd58ec75d49c9]) successfully deleted
I20260810 23:30:22.350937 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 11179
W20260810 23:30:22.389500 11063 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.175.65:37063 (error 108)
I20260810 23:30:22.390601 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 11388
I20260810 23:30:22.392091 11792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:22.391526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:37063 (local address 127.0.0.1:38728)
0810 23:30:22.391791 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:22.391923 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:37063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
W20260810 23:30:22.392900 11063 catalog_manager.cc:4584] TS 3ec872414ab8401d938bcd16797ea5c1 (127.14.175.65:37063): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 35832babe63d4dc7bedce1994c9c3ed4: Network error: Client connection negotiation failed: client connection to 127.14.175.65:37063: connect: Connection refused (error 111)
I20260810 23:30:22.399554 11802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:22.391780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51618 (local address 127.14.175.66:35465)
0810 23:30:22.392360 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:22.392367 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:22.392387 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:30:22.392453 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:22.392458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:22.392561 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:22.392676 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:22.393257 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:22.394436 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:22.395779 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:22.396447 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:22.396579 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:22.397618 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:22.397630 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:22.397651 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:22.398632 (+ 981us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:22.398700 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:22.398727 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:22.398847 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:22.399005 (+ 158us) server_negotiation.cc:299] Negotiation successful
0810 23:30:22.399136 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455}
I20260810 23:30:22.400386 11564 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:30:22.447032 11792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:22.446414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:37063 (local address 127.0.0.1:38740)
0810 23:30:22.446693 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:22.446813 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:37063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260810 23:30:22.448241 11792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:22.447803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:37063 (local address 127.0.0.1:38742)
0810 23:30:22.447964 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:22.448057 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:37063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 23:30:22.508852 11792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:22.508245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:37063 (local address 127.0.0.1:38750)
0810 23:30:22.508487 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:22.508611 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:37063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260810 23:30:22.510016 11792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:22.509591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:37063 (local address 127.0.0.1:38766)
0810 23:30:22.509763 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:22.509849 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:37063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 23:30:22.513685 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 11388
I20260810 23:30:22.538174 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 11578
I20260810 23:30:22.545454 11797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:22.538961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43254 (local address 127.14.175.67:40777)
0810 23:30:22.539151 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:22.539157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:22.539211 (+ 54us) server_negotiation.cc:407] Connection header received
0810 23:30:22.539426 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:22.539430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:22.539531 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:22.539620 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:22.540104 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:22.540900 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:22.542338 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:22.542751 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:22.542872 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:22.544033 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:22.544043 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:22.544059 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:22.544664 (+ 605us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:22.544718 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:22.544740 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:22.544855 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:22.544991 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 23:30:22.545114 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 23:30:22.546204 11766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:30:22.590950 11792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:22.590334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:37063 (local address 127.0.0.1:38770)
0810 23:30:22.590613 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:22.590740 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:37063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260810 23:30:22.592307 11792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:22.591823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:37063 (local address 127.0.0.1:38776)
0810 23:30:22.592058 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:22.592153 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:37063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260810 23:30:22.636372 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 11578
I20260810 23:30:22.661321 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 11044
I20260810 23:30:22.662463 11134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:30:22.775456 11792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:22.774888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:37063 (local address 127.0.0.1:38784)
0810 23:30:22.775113 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:22.775244 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:37063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260810 23:30:22.775537 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 11044
2026-08-10T23:30:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2987 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:30:22 dist-test-slave-rmhg krb5kdc[11862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:30:22 dist-test-slave-rmhg krb5kdc[11862](info): set up 2 sockets
Aug 10 23:30:22 dist-test-slave-rmhg krb5kdc[11862](info): commencing operation
krb5kdc: starting...
W20260810 23:30:24.018416 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.193s user 0.003s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:30:24 dist-test-slave-rmhg krb5kdc[11862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:30:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:30:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
I20260810 23:30:24.334554 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:38699
--webserver_interface=127.14.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:36591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:38699
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:30:24.693238 11886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:30:24.693879 11886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:30:24.694103 11886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:30:24.703927 11886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:30:24.704089 11886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:30:24.704185 11886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:30:24.704253 11886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:30:24.704303 11886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:30:24.718238 11886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:36591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:38699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:38699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:30:24.720157 11886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:30:24.722030 11886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:30:24.736660 11893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:24.736629 11897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:24.737650 11895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:30:24.737850 11886 server_base.cc:1034] running on GCE node
I20260810 23:30:24.738911 11886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:30:24.740993 11886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:30:24.742252 11886 hybrid_clock.cc:648] HybridClock initialized: now 1786404624742169 us; error 69 us; skew 500 ppm
I20260810 23:30:24.742643 11886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:30:24 dist-test-slave-rmhg krb5kdc[11862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404624, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:30:24.746487 11886 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:30:24.748335 11886 webserver.cc:460] Webserver started at http://127.14.175.126:38165/ using document root <none> and password file <none>
I20260810 23:30:24.749082 11886 fs_manager.cc:359] Metadata directory not provided
I20260810 23:30:24.749305 11886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:30:24.749650 11886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:30:24.753804 11886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "e6d389901e914a7fab754036a0e860b8"
format_stamp: "Formatted at 2026-08-10 23:30:24 on dist-test-slave-rmhg"
server_key: "158354e901421379effe08f61ebd6d2f"
server_key_iv: "0f275afec13a8651389f8e8db614b31a"
server_key_version: "encryptionkey@0"
I20260810 23:30:24.754626 11886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "e6d389901e914a7fab754036a0e860b8"
format_stamp: "Formatted at 2026-08-10 23:30:24 on dist-test-slave-rmhg"
server_key: "158354e901421379effe08f61ebd6d2f"
server_key_iv: "0f275afec13a8651389f8e8db614b31a"
server_key_version: "encryptionkey@0"
I20260810 23:30:24.760720 11886 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260810 23:30:24.764732 11910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:24.766232 11886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:30:24.766420 11886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "e6d389901e914a7fab754036a0e860b8"
format_stamp: "Formatted at 2026-08-10 23:30:24 on dist-test-slave-rmhg"
server_key: "158354e901421379effe08f61ebd6d2f"
server_key_iv: "0f275afec13a8651389f8e8db614b31a"
server_key_version: "encryptionkey@0"
I20260810 23:30:24.766626 11886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:30:24.785588 11886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:30:24.790320 11886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:30:24.790617 11886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:30:24.809904 11886 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:38699
I20260810 23:30:24.809916 11982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:38699 every 8 connection(s)
I20260810 23:30:24.811434 11886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:30:24.815706 11984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:30:24.819413 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 11886
I20260810 23:30:24.819607 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:30:24.820048 15037 external_mini_cluster.cc:1428] Setting key 3fa97ec32b683953c5d422dc34974705
I20260810 23:30:24.829470 11984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6d389901e914a7fab754036a0e860b8: Bootstrap starting.
I20260810 23:30:24.833681 11984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6d389901e914a7fab754036a0e860b8: Neither blocks nor log segments found. Creating new log.
I20260810 23:30:24.835086 11984 log.cc:824] T 00000000000000000000000000000000 P e6d389901e914a7fab754036a0e860b8: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:30:24 dist-test-slave-rmhg krb5kdc[11862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:30:24.838739 11984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6d389901e914a7fab754036a0e860b8: No bootstrap required, opened a new log
I20260810 23:30:24.845625 11989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:24.822088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33944 (local address 127.14.175.126:38699)
0810 23:30:24.822926 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:24.822939 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:24.822985 (+ 46us) server_negotiation.cc:407] Connection header received
0810 23:30:24.824884 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:24.824916 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:24.825446 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:24.825826 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:24.827804 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:24.829195 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:24.831196 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:24.831949 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:24.832102 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:24.837743 (+ 5641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:24.837752 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:24.837769 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:30:24.837814 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:24.840906 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:24.841889 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:24.841898 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:24.841906 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:24.842114 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:24.842556 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:24.842560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:24.842562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:24.843167 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:24.843431 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:24.843945 (+ 514us) server_negotiation.cc:299] Negotiation successful
0810 23:30:24.844390 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":373,"threads_started":1}
I20260810 23:30:24.847879 11984 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6d389901e914a7fab754036a0e860b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6d389901e914a7fab754036a0e860b8" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38699 } }
I20260810 23:30:24.848263 11984 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6d389901e914a7fab754036a0e860b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:30:24.848376 11984 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6d389901e914a7fab754036a0e860b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6d389901e914a7fab754036a0e860b8, State: Initialized, Role: FOLLOWER
I20260810 23:30:24.848954 11984 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6d389901e914a7fab754036a0e860b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6d389901e914a7fab754036a0e860b8" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38699 } }
I20260810 23:30:24.849215 11984 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6d389901e914a7fab754036a0e860b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:30:24.849354 11984 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6d389901e914a7fab754036a0e860b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:30:24.849520 11984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6d389901e914a7fab754036a0e860b8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:30:24.851781 11984 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6d389901e914a7fab754036a0e860b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6d389901e914a7fab754036a0e860b8" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38699 } }
I20260810 23:30:24.852284 11984 leader_election.cc:302] T 00000000000000000000000000000000 P e6d389901e914a7fab754036a0e860b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6d389901e914a7fab754036a0e860b8; no voters:
I20260810 23:30:24.853025 11984 leader_election.cc:288] T 00000000000000000000000000000000 P e6d389901e914a7fab754036a0e860b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:30:24.853144 11992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6d389901e914a7fab754036a0e860b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:30:24.853864 11992 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6d389901e914a7fab754036a0e860b8 [term 1 LEADER]: Becoming Leader. State: Replica: e6d389901e914a7fab754036a0e860b8, State: Running, Role: LEADER
I20260810 23:30:24.854362 11992 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6d389901e914a7fab754036a0e860b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6d389901e914a7fab754036a0e860b8" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38699 } }
I20260810 23:30:24.855155 11984 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6d389901e914a7fab754036a0e860b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:30:24.857234 11999 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6d389901e914a7fab754036a0e860b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6d389901e914a7fab754036a0e860b8. Latest consensus state: current_term: 1 leader_uuid: "e6d389901e914a7fab754036a0e860b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6d389901e914a7fab754036a0e860b8" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38699 } } }
I20260810 23:30:24.857681 11999 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6d389901e914a7fab754036a0e860b8 [sys.catalog]: This master's current role is: LEADER
I20260810 23:30:24.857302 11995 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6d389901e914a7fab754036a0e860b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6d389901e914a7fab754036a0e860b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6d389901e914a7fab754036a0e860b8" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 38699 } } }
I20260810 23:30:24.865085 11995 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6d389901e914a7fab754036a0e860b8 [sys.catalog]: This master's current role is: LEADER
I20260810 23:30:24.865447 12010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:30:24.870179 12010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:30:24.879248 12010 catalog_manager.cc:1347] Generated new cluster ID: 840fd0b4788b45189560b42c37e4d64c
I20260810 23:30:24.879383 12010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:30:24.901193 12010 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:30:24.903349 12010 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:30:24.913713 12010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6d389901e914a7fab754036a0e860b8: Generated new TSK 0
I20260810 23:30:24.914956 12010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
I20260810 23:30:24.997753 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.65:0
--local_ip_for_outbound_sockets=127.14.175.65
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:38699
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:36591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:30:25.364401 12031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:30:25.364732 12031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:30:25.364874 12031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:30:25.372542 12031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:30:25.372691 12031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:30:25.372819 12031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.65
I20260810 23:30:25.383383 12031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:36591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:38699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.65
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:30:25.384800 12031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:30:25.386242 12031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:30:25.400032 12036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:25.399948 12038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:25.399976 12043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:30:25.401279 12031 server_base.cc:1034] running on GCE node
I20260810 23:30:25.402040 12031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:30:25.403153 12031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:30:25.404390 12031 hybrid_clock.cc:648] HybridClock initialized: now 1786404625404332 us; error 55 us; skew 500 ppm
I20260810 23:30:25.404771 12031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:30:25 dist-test-slave-rmhg krb5kdc[11862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404625, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:30:25.413442 12031 init.cc:377] Logged in from keytab as kudu/127.14.175.65@KRBTEST.COM (short username kudu)
I20260810 23:30:25.415089 12031 webserver.cc:460] Webserver started at http://127.14.175.65:37671/ using document root <none> and password file <none>
I20260810 23:30:25.415834 12031 fs_manager.cc:359] Metadata directory not provided
I20260810 23:30:25.415963 12031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:30:25.416287 12031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:30:25.419490 12031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-0/data/instance:
uuid: "6117a0cd17234332b505b72d4001e76f"
format_stamp: "Formatted at 2026-08-10 23:30:25 on dist-test-slave-rmhg"
server_key: "e3074d080cb76461b9498a34a8f0448f"
server_key_iv: "81042b93d33a76ef954e57af9ea2ffc0"
server_key_version: "encryptionkey@0"
I20260810 23:30:25.420305 12031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance:
uuid: "6117a0cd17234332b505b72d4001e76f"
format_stamp: "Formatted at 2026-08-10 23:30:25 on dist-test-slave-rmhg"
server_key: "e3074d080cb76461b9498a34a8f0448f"
server_key_iv: "81042b93d33a76ef954e57af9ea2ffc0"
server_key_version: "encryptionkey@0"
I20260810 23:30:25.425998 12031 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260810 23:30:25.429821 12055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:25.431314 12031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:30:25.431470 12031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-0/wal
uuid: "6117a0cd17234332b505b72d4001e76f"
format_stamp: "Formatted at 2026-08-10 23:30:25 on dist-test-slave-rmhg"
server_key: "e3074d080cb76461b9498a34a8f0448f"
server_key_iv: "81042b93d33a76ef954e57af9ea2ffc0"
server_key_version: "encryptionkey@0"
I20260810 23:30:25.431665 12031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:30:25.460593 12031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:30:25.465476 12031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:30:25.465760 12031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:30:25.466871 12031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:30:25.468302 12031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:30:25.468389 12031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:25.468483 12031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:30:25.468528 12031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:25.501094 12031 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.65:41789
I20260810 23:30:25.501207 12216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.65:41789 every 8 connection(s)
I20260810 23:30:25.502483 12031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
I20260810 23:30:25.511224 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 12031
I20260810 23:30:25.511430 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance
I20260810 23:30:25.511888 15037 external_mini_cluster.cc:1428] Setting key c92d6722269d4e4b9363a01e82da6ea5
Aug 10 23:30:25 dist-test-slave-rmhg krb5kdc[11862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404625, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:30:25.522984 12219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:25.505035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:58451 (local address 127.14.175.126:38699)
0810 23:30:25.505548 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:25.505554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:25.506548 (+ 994us) server_negotiation.cc:407] Connection header received
0810 23:30:25.508406 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:25.508411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:25.508485 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:25.508585 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:25.510201 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:25.511057 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:25.512781 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:25.513244 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:25.513367 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:25.517199 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:25.517207 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:25.517212 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:30:25.517254 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:25.519740 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:25.520522 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:25.520527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:25.520531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:25.520665 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:25.521109 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:25.521114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:25.521116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:25.521431 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:25.521576 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:25.522356 (+ 780us) server_negotiation.cc:299] Negotiation successful
0810 23:30:25.522613 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":251,"threads_started":1}
I20260810 23:30:25.524019 12221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:25.505628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:38699 (local address 127.14.175.65:58451)
0810 23:30:25.506371 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:25.506411 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:25.508119 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:25.508788 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:25.508797 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:25.509409 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:30:25.510022 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:25.510041 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:25.511199 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:25.511203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:25.512627 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:25.512645 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:25.513525 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:25.513534 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:25.513833 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:25.514532 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:30:25.514558 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:30:25.516908 (+ 2350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:30:25.519897 (+ 2989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:30:25.519908 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:30:25.519929 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:30:25.520394 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:30:25.520775 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:30:25.520781 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:30:25.520785 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:30:25.520991 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:30:25.521566 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:30:25.521575 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:30:25.521956 (+ 381us) client_negotiation.cc:769] Sending connection context
0810 23:30:25.522231 (+ 275us) client_negotiation.cc:240] Negotiation successful
0810 23:30:25.522825 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":282,"threads_started":1}
I20260810 23:30:25.525748 12217 heartbeater.cc:347] Connected to a master server at 127.14.175.126:38699
I20260810 23:30:25.526098 12217 heartbeater.cc:464] Registering TS with master...
I20260810 23:30:25.526901 12217 heartbeater.cc:511] Master 127.14.175.126:38699 requested a full tablet report, sending...
I20260810 23:30:25.529208 11937 ts_manager.cc:194] Registered new tserver with Master: 6117a0cd17234332b505b72d4001e76f (127.14.175.65:41789)
I20260810 23:30:25.530773 11937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.65@KRBTEST.COM'} at 127.14.175.65:58451
WARNING: no policy specified for kudu/127.14.175.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
I20260810 23:30:25.578572 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.66:0
--local_ip_for_outbound_sockets=127.14.175.66
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:38699
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:36591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:30:25.928983 12226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:30:25.929328 12226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:30:25.929504 12226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:30:25.936869 12226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:30:25.936998 12226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:30:25.937176 12226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.66
I20260810 23:30:25.947110 12226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:36591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:38699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.66
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:30:25.948459 12226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:30:25.949846 12226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:30:25.962723 12232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:25.962671 12234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:25.962642 12239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:30:25.963510 12226 server_base.cc:1034] running on GCE node
I20260810 23:30:25.964143 12226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:30:25.965174 12226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:30:25.966398 12226 hybrid_clock.cc:648] HybridClock initialized: now 1786404625966343 us; error 44 us; skew 500 ppm
I20260810 23:30:25.966739 12226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:30:25 dist-test-slave-rmhg krb5kdc[11862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404625, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:30:25.969974 12226 init.cc:377] Logged in from keytab as kudu/127.14.175.66@KRBTEST.COM (short username kudu)
I20260810 23:30:25.971522 12226 webserver.cc:460] Webserver started at http://127.14.175.66:33347/ using document root <none> and password file <none>
I20260810 23:30:25.972224 12226 fs_manager.cc:359] Metadata directory not provided
I20260810 23:30:25.972352 12226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:30:25.972666 12226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:30:25.975713 12226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-1/data/instance:
uuid: "66275db21bf14f86bec69e88109441a3"
format_stamp: "Formatted at 2026-08-10 23:30:25 on dist-test-slave-rmhg"
server_key: "8ba0fb77087d35ab95da333661fad704"
server_key_iv: "9d96cc110c81ce5c95400e1794b9f1fd"
server_key_version: "encryptionkey@0"
I20260810 23:30:25.976490 12226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance:
uuid: "66275db21bf14f86bec69e88109441a3"
format_stamp: "Formatted at 2026-08-10 23:30:25 on dist-test-slave-rmhg"
server_key: "8ba0fb77087d35ab95da333661fad704"
server_key_iv: "9d96cc110c81ce5c95400e1794b9f1fd"
server_key_version: "encryptionkey@0"
I20260810 23:30:25.982046 12226 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260810 23:30:25.985587 12251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:25.986984 12226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:30:25.987146 12226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-1/wal
uuid: "66275db21bf14f86bec69e88109441a3"
format_stamp: "Formatted at 2026-08-10 23:30:25 on dist-test-slave-rmhg"
server_key: "8ba0fb77087d35ab95da333661fad704"
server_key_iv: "9d96cc110c81ce5c95400e1794b9f1fd"
server_key_version: "encryptionkey@0"
I20260810 23:30:25.987350 12226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:30:26.004855 12226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:30:26.008994 12226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:30:26.009318 12226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:30:26.010363 12226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:30:26.011751 12226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:30:26.011823 12226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:26.011914 12226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:30:26.011957 12226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:26.042868 12226 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.66:34931
I20260810 23:30:26.042933 12410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.66:34931 every 8 connection(s)
I20260810 23:30:26.044272 12226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
I20260810 23:30:26.050521 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 12226
I20260810 23:30:26.050746 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance
I20260810 23:30:26.051307 15037 external_mini_cluster.cc:1428] Setting key a18ad15d22571f81bff0191c4bd0fd2e
Aug 10 23:30:26 dist-test-slave-rmhg krb5kdc[11862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404625, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:30:26.065205 12417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:26.046969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:57541 (local address 127.14.175.126:38699)
0810 23:30:26.047436 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:26.047442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:26.048341 (+ 899us) server_negotiation.cc:407] Connection header received
0810 23:30:26.050331 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:26.050336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:26.050413 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:26.050502 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:26.052553 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:26.053357 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:26.054944 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:26.055374 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:26.055490 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:26.059269 (+ 3779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:26.059275 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:26.059278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:30:26.059315 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:26.061960 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:26.062738 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:26.062743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:26.062745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:26.062876 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:26.063362 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:26.063365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:26.063368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:26.063643 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:26.063780 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:26.064521 (+ 741us) server_negotiation.cc:299] Negotiation successful
0810 23:30:26.064770 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":246,"threads_started":1}
I20260810 23:30:26.066231 12422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:26.047419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:38699 (local address 127.14.175.66:57541)
0810 23:30:26.048162 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:26.048207 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:26.049966 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:26.050815 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:26.050827 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:26.051570 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:30:26.052372 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:26.052399 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:26.053516 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:26.053520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:26.054784 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:26.054798 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:26.055644 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:26.055653 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:26.055881 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:26.056548 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:30:26.056577 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:30:26.059002 (+ 2425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:30:26.062127 (+ 3125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:30:26.062136 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:30:26.062155 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:30:26.062606 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:30:26.062990 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:30:26.062995 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:30:26.062999 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:30:26.063254 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:30:26.063773 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:30:26.063782 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:30:26.064149 (+ 367us) client_negotiation.cc:769] Sending connection context
0810 23:30:26.064408 (+ 259us) client_negotiation.cc:240] Negotiation successful
0810 23:30:26.064993 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":290,"threads_started":1}
I20260810 23:30:26.067771 12412 heartbeater.cc:347] Connected to a master server at 127.14.175.126:38699
I20260810 23:30:26.068111 12412 heartbeater.cc:464] Registering TS with master...
I20260810 23:30:26.069061 12412 heartbeater.cc:511] Master 127.14.175.126:38699 requested a full tablet report, sending...
I20260810 23:30:26.070716 11937 ts_manager.cc:194] Registered new tserver with Master: 66275db21bf14f86bec69e88109441a3 (127.14.175.66:34931)
I20260810 23:30:26.071589 11937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.66@KRBTEST.COM'} at 127.14.175.66:57541
WARNING: no policy specified for kudu/127.14.175.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
I20260810 23:30:26.120147 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.67:0
--local_ip_for_outbound_sockets=127.14.175.67
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:38699
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:36591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:30:26.476976 12425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:30:26.477347 12425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:30:26.477502 12425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:30:26.485008 12425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:30:26.485139 12425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:30:26.485321 12425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.67
I20260810 23:30:26.495652 12425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:36591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:38699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.67
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:30:26.497041 12425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:30:26.498436 12425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:30:26.513099 12433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:26.513141 12434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:26.513202 12436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:30:26.514153 12425 server_base.cc:1034] running on GCE node
I20260810 23:30:26.514781 12425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:30:26.515702 12425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:30:26.516901 12425 hybrid_clock.cc:648] HybridClock initialized: now 1786404626516846 us; error 42 us; skew 500 ppm
I20260810 23:30:26.517246 12425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:30:26 dist-test-slave-rmhg krb5kdc[11862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404626, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:30:26.520349 12425 init.cc:377] Logged in from keytab as kudu/127.14.175.67@KRBTEST.COM (short username kudu)
I20260810 23:30:26.521880 12425 webserver.cc:460] Webserver started at http://127.14.175.67:36969/ using document root <none> and password file <none>
I20260810 23:30:26.522627 12425 fs_manager.cc:359] Metadata directory not provided
I20260810 23:30:26.522750 12425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:30:26.523051 12425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:30:26.526153 12425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-2/data/instance:
uuid: "ca31005266f8418db3a0a363707bf6ad"
format_stamp: "Formatted at 2026-08-10 23:30:26 on dist-test-slave-rmhg"
server_key: "b69e1bda1cadf5a2ff31769f1fe1659d"
server_key_iv: "54b72ff3986b08135f964ec03c5233a4"
server_key_version: "encryptionkey@0"
I20260810 23:30:26.526939 12425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance:
uuid: "ca31005266f8418db3a0a363707bf6ad"
format_stamp: "Formatted at 2026-08-10 23:30:26 on dist-test-slave-rmhg"
server_key: "b69e1bda1cadf5a2ff31769f1fe1659d"
server_key_iv: "54b72ff3986b08135f964ec03c5233a4"
server_key_version: "encryptionkey@0"
I20260810 23:30:26.532419 12425 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260810 23:30:26.534087 12217 heartbeater.cc:503] Master 127.14.175.126:38699 was elected leader, sending a full tablet report...
I20260810 23:30:26.536659 12446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:26.538204 12425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:30:26.538371 12425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-2/wal
uuid: "ca31005266f8418db3a0a363707bf6ad"
format_stamp: "Formatted at 2026-08-10 23:30:26 on dist-test-slave-rmhg"
server_key: "b69e1bda1cadf5a2ff31769f1fe1659d"
server_key_iv: "54b72ff3986b08135f964ec03c5233a4"
server_key_version: "encryptionkey@0"
I20260810 23:30:26.538565 12425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:30:26.565687 12425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:30:26.570523 12425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:30:26.570842 12425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:30:26.571858 12425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:30:26.573279 12425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:30:26.573369 12425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:26.573490 12425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:30:26.573545 12425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:26.604195 12425 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.67:43219
I20260810 23:30:26.604249 12623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.67:43219 every 8 connection(s)
I20260810 23:30:26.605551 12425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
I20260810 23:30:26.614300 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 12425
I20260810 23:30:26.614487 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance
I20260810 23:30:26.614949 15037 external_mini_cluster.cc:1428] Setting key 9cb431f03687df88d51b5cb535cb4fb7
Aug 10 23:30:26 dist-test-slave-rmhg krb5kdc[11862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404626, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:30:26.625476 12627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:26.608004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:36329 (local address 127.14.175.126:38699)
0810 23:30:26.608478 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:26.608485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:26.609485 (+ 1000us) server_negotiation.cc:407] Connection header received
0810 23:30:26.611351 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:26.611356 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:26.611432 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:26.611531 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:26.613215 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:26.614120 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:26.616005 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:26.616569 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:26.616698 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:26.620302 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:26.620307 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:26.620311 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:30:26.620354 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:26.622756 (+ 2402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:26.623454 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:26.623458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:26.623460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:26.623583 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:26.623924 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:26.623927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:26.623929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:26.624199 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:26.624324 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:26.624965 (+ 641us) server_negotiation.cc:299] Negotiation successful
0810 23:30:26.625138 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":219,"threads_started":1}
I20260810 23:30:26.626540 12629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:26.608533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:38699 (local address 127.14.175.67:36329)
0810 23:30:26.609307 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:26.609349 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:26.611064 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:26.611790 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:26.611799 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:26.612404 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:30:26.613014 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:26.613033 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:26.614321 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:26.614325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:26.615852 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:26.615875 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:26.616822 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:26.616831 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:26.617095 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:26.617800 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:30:26.617834 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:30:26.620076 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:30:26.622888 (+ 2812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:30:26.622896 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:30:26.622910 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:30:26.623363 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:30:26.623661 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:30:26.623664 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:30:26.623666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:30:26.623846 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:30:26.624298 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:30:26.624304 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:30:26.624649 (+ 345us) client_negotiation.cc:769] Sending connection context
0810 23:30:26.624903 (+ 254us) client_negotiation.cc:240] Negotiation successful
0810 23:30:26.625409 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":279,"threads_started":1}
I20260810 23:30:26.627987 12624 heartbeater.cc:347] Connected to a master server at 127.14.175.126:38699
I20260810 23:30:26.628319 12624 heartbeater.cc:464] Registering TS with master...
I20260810 23:30:26.629205 12624 heartbeater.cc:511] Master 127.14.175.126:38699 requested a full tablet report, sending...
I20260810 23:30:26.630884 11937 ts_manager.cc:194] Registered new tserver with Master: ca31005266f8418db3a0a363707bf6ad (127.14.175.67:43219)
I20260810 23:30:26.631693 11937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.67@KRBTEST.COM'} at 127.14.175.67:36329
I20260810 23:30:26.643373 15037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:30:26.660809 12627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:26.647540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33956 (local address 127.14.175.126:38699)
0810 23:30:26.647770 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:26.647778 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:26.648144 (+ 366us) server_negotiation.cc:407] Connection header received
0810 23:30:26.648292 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:26.648297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:26.648367 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:26.648450 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:26.650003 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:26.650770 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:26.652044 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:26.652464 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:26.652560 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:26.655403 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:26.655409 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:26.655413 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:30:26.655451 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:26.658003 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:26.658864 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:26.658868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:26.658871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:26.659006 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:26.659433 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:26.659437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:26.659439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:26.659750 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:26.659953 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:26.660355 (+ 402us) server_negotiation.cc:299] Negotiation successful
0810 23:30:26.660502 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 23:30:26.664822 11937 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:30:26.671813 11937 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:30:26.707610 12648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:26.684926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:34931 (local address 127.0.0.1:50686)
0810 23:30:26.696353 (+ 11427us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:26.696409 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:26.696624 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:26.699310 (+ 2686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:26.699320 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:26.699372 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:30:26.699919 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:26.699934 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:26.701925 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:26.701931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:26.704758 (+ 2827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:26.704778 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:26.706659 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:26.706666 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:26.706830 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:26.706854 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 23:30:26.706915 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 23:30:26.707152 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11091,"mutex_wait_us":104,"spinlock_wait_cycles":2304,"thread_start_us":12796,"threads_started":1}
I20260810 23:30:26.707686 12649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:26.686006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51374 (local address 127.14.175.67:43219)
0810 23:30:26.696808 (+ 10802us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:26.696817 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:26.698317 (+ 1500us) server_negotiation.cc:407] Connection header received
0810 23:30:26.698444 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:26.698451 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:26.698753 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:26.698965 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:30:26.700884 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:26.702532 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:26.705202 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:26.706283 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:26.706414 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:26.706669 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:26.707119 (+ 450us) server_negotiation.cc:299] Negotiation successful
0810 23:30:26.707336 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10603,"thread_start_us":10432,"threads_started":1}
I20260810 23:30:26.708781 12651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:26.698729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:41789 (local address 127.0.0.1:56164)
0810 23:30:26.700826 (+ 2097us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:26.700847 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:26.700977 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:26.701672 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:26.701677 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:26.701703 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:30:26.702163 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:26.702181 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:26.704015 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:26.704024 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:26.705952 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:26.705968 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:26.708105 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:26.708110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:26.708235 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:26.708260 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 23:30:26.708319 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 23:30:26.708417 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1786,"mutex_wait_us":8,"thread_start_us":1357,"threads_started":1}
I20260810 23:30:26.708828 12646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:26.685391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:43219 (local address 127.0.0.1:51374)
0810 23:30:26.698028 (+ 12637us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:26.698049 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:26.698184 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:26.699323 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:26.699329 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:26.699381 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:30:26.700593 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:26.700610 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:26.702721 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:26.702728 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:26.705013 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:26.705024 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:26.706493 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:26.706501 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:26.706739 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:26.706802 (+ 63us) client_negotiation.cc:769] Sending connection context
0810 23:30:26.706938 (+ 136us) client_negotiation.cc:240] Negotiation successful
0810 23:30:26.708383 (+ 1445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12508,"mutex_wait_us":1316,"spinlock_wait_cycles":6912,"thread_start_us":11452,"threads_started":1}
I20260810 23:30:26.709612 12647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:26.684981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50686 (local address 127.14.175.66:34931)
0810 23:30:26.698581 (+ 13600us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:26.698591 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:26.698620 (+ 29us) server_negotiation.cc:407] Connection header received
0810 23:30:26.698740 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:26.698747 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:26.699007 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:26.699722 (+ 715us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:30:26.700292 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:26.701729 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:26.704966 (+ 3237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:26.706407 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:26.708787 (+ 2380us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:26.708987 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:26.709079 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 23:30:26.709292 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13450,"thread_start_us":13335,"threads_started":1}
I20260810 23:30:26.709731 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:26.698532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56164 (local address 127.14.175.65:41789)
0810 23:30:26.700776 (+ 2244us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:26.700784 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:26.700957 (+ 173us) server_negotiation.cc:407] Connection header received
0810 23:30:26.701134 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:26.701142 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:26.701452 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:26.701660 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:30:26.702382 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:26.703865 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:26.706141 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:26.707571 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:26.707726 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:26.709099 (+ 1373us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:26.709222 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 23:30:26.709404 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2063,"thread_start_us":1889,"threads_started":1}
I20260810 23:30:26.712937 12524 tablet_service.cc:1467] Processing CreateTablet for tablet a63ea7226d914bbcafb98cba5585cd98 (DEFAULT_TABLE table=test-table [id=d28e57600ab04346b79e21ba784539c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:30:26.713440 12316 tablet_service.cc:1467] Processing CreateTablet for tablet a63ea7226d914bbcafb98cba5585cd98 (DEFAULT_TABLE table=test-table [id=d28e57600ab04346b79e21ba784539c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:30:26.713441 12123 tablet_service.cc:1467] Processing CreateTablet for tablet a63ea7226d914bbcafb98cba5585cd98 (DEFAULT_TABLE table=test-table [id=d28e57600ab04346b79e21ba784539c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:30:26.714371 12524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a63ea7226d914bbcafb98cba5585cd98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:30:26.714859 12123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a63ea7226d914bbcafb98cba5585cd98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:30:26.714857 12316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a63ea7226d914bbcafb98cba5585cd98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:30:26.728041 12652 tablet_bootstrap.cc:492] T a63ea7226d914bbcafb98cba5585cd98 P ca31005266f8418db3a0a363707bf6ad: Bootstrap starting.
I20260810 23:30:26.730188 12655 tablet_bootstrap.cc:492] T a63ea7226d914bbcafb98cba5585cd98 P 66275db21bf14f86bec69e88109441a3: Bootstrap starting.
I20260810 23:30:26.731848 12656 tablet_bootstrap.cc:492] T a63ea7226d914bbcafb98cba5585cd98 P 6117a0cd17234332b505b72d4001e76f: Bootstrap starting.
I20260810 23:30:26.732441 12652 tablet_bootstrap.cc:654] T a63ea7226d914bbcafb98cba5585cd98 P ca31005266f8418db3a0a363707bf6ad: Neither blocks nor log segments found. Creating new log.
I20260810 23:30:26.733968 12655 tablet_bootstrap.cc:654] T a63ea7226d914bbcafb98cba5585cd98 P 66275db21bf14f86bec69e88109441a3: Neither blocks nor log segments found. Creating new log.
I20260810 23:30:26.733989 12652 log.cc:824] T a63ea7226d914bbcafb98cba5585cd98 P ca31005266f8418db3a0a363707bf6ad: Log is configured to *not* fsync() on all Append() calls
I20260810 23:30:26.734997 12656 tablet_bootstrap.cc:654] T a63ea7226d914bbcafb98cba5585cd98 P 6117a0cd17234332b505b72d4001e76f: Neither blocks nor log segments found. Creating new log.
I20260810 23:30:26.735388 12655 log.cc:824] T a63ea7226d914bbcafb98cba5585cd98 P 66275db21bf14f86bec69e88109441a3: Log is configured to *not* fsync() on all Append() calls
I20260810 23:30:26.736243 12656 log.cc:824] T a63ea7226d914bbcafb98cba5585cd98 P 6117a0cd17234332b505b72d4001e76f: Log is configured to *not* fsync() on all Append() calls
I20260810 23:30:26.739230 12655 tablet_bootstrap.cc:492] T a63ea7226d914bbcafb98cba5585cd98 P 66275db21bf14f86bec69e88109441a3: No bootstrap required, opened a new log
I20260810 23:30:26.739288 12652 tablet_bootstrap.cc:492] T a63ea7226d914bbcafb98cba5585cd98 P ca31005266f8418db3a0a363707bf6ad: No bootstrap required, opened a new log
I20260810 23:30:26.739548 12655 ts_tablet_manager.cc:1397] T a63ea7226d914bbcafb98cba5585cd98 P 66275db21bf14f86bec69e88109441a3: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260810 23:30:26.739562 12652 ts_tablet_manager.cc:1397] T a63ea7226d914bbcafb98cba5585cd98 P ca31005266f8418db3a0a363707bf6ad: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260810 23:30:26.741523 12656 tablet_bootstrap.cc:492] T a63ea7226d914bbcafb98cba5585cd98 P 6117a0cd17234332b505b72d4001e76f: No bootstrap required, opened a new log
I20260810 23:30:26.741859 12656 ts_tablet_manager.cc:1397] T a63ea7226d914bbcafb98cba5585cd98 P 6117a0cd17234332b505b72d4001e76f: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20260810 23:30:26.747772 12652 raft_consensus.cc:348] T a63ea7226d914bbcafb98cba5585cd98 P ca31005266f8418db3a0a363707bf6ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66275db21bf14f86bec69e88109441a3" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 34931 } } peers { permanent_uuid: "6117a0cd17234332b505b72d4001e76f" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 41789 } } peers { permanent_uuid: "ca31005266f8418db3a0a363707bf6ad" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43219 } }
I20260810 23:30:26.748164 12652 raft_consensus.cc:374] T a63ea7226d914bbcafb98cba5585cd98 P ca31005266f8418db3a0a363707bf6ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:30:26.748260 12652 raft_consensus.cc:729] T a63ea7226d914bbcafb98cba5585cd98 P ca31005266f8418db3a0a363707bf6ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca31005266f8418db3a0a363707bf6ad, State: Initialized, Role: FOLLOWER
I20260810 23:30:26.748905 12652 consensus_queue.cc:260] T a63ea7226d914bbcafb98cba5585cd98 P ca31005266f8418db3a0a363707bf6ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66275db21bf14f86bec69e88109441a3" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 34931 } } peers { permanent_uuid: "6117a0cd17234332b505b72d4001e76f" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 41789 } } peers { permanent_uuid: "ca31005266f8418db3a0a363707bf6ad" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43219 } }
I20260810 23:30:26.750056 12624 heartbeater.cc:503] Master 127.14.175.126:38699 was elected leader, sending a full tablet report...
I20260810 23:30:26.750259 12655 raft_consensus.cc:348] T a63ea7226d914bbcafb98cba5585cd98 P 66275db21bf14f86bec69e88109441a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66275db21bf14f86bec69e88109441a3" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 34931 } } peers { permanent_uuid: "6117a0cd17234332b505b72d4001e76f" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 41789 } } peers { permanent_uuid: "ca31005266f8418db3a0a363707bf6ad" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43219 } }
I20260810 23:30:26.750730 12655 raft_consensus.cc:374] T a63ea7226d914bbcafb98cba5585cd98 P 66275db21bf14f86bec69e88109441a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:30:26.750826 12655 raft_consensus.cc:729] T a63ea7226d914bbcafb98cba5585cd98 P 66275db21bf14f86bec69e88109441a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66275db21bf14f86bec69e88109441a3, State: Initialized, Role: FOLLOWER
I20260810 23:30:26.750900 12652 ts_tablet_manager.cc:1428] T a63ea7226d914bbcafb98cba5585cd98 P ca31005266f8418db3a0a363707bf6ad: Time spent starting tablet: real 0.011s user 0.010s sys 0.004s
I20260810 23:30:26.751571 12655 consensus_queue.cc:260] T a63ea7226d914bbcafb98cba5585cd98 P 66275db21bf14f86bec69e88109441a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66275db21bf14f86bec69e88109441a3" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 34931 } } peers { permanent_uuid: "6117a0cd17234332b505b72d4001e76f" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 41789 } } peers { permanent_uuid: "ca31005266f8418db3a0a363707bf6ad" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43219 } }
I20260810 23:30:26.752470 12656 raft_consensus.cc:348] T a63ea7226d914bbcafb98cba5585cd98 P 6117a0cd17234332b505b72d4001e76f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66275db21bf14f86bec69e88109441a3" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 34931 } } peers { permanent_uuid: "6117a0cd17234332b505b72d4001e76f" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 41789 } } peers { permanent_uuid: "ca31005266f8418db3a0a363707bf6ad" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43219 } }
I20260810 23:30:26.752970 12656 raft_consensus.cc:374] T a63ea7226d914bbcafb98cba5585cd98 P 6117a0cd17234332b505b72d4001e76f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:30:26.753058 12656 raft_consensus.cc:729] T a63ea7226d914bbcafb98cba5585cd98 P 6117a0cd17234332b505b72d4001e76f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6117a0cd17234332b505b72d4001e76f, State: Initialized, Role: FOLLOWER
I20260810 23:30:26.753774 12656 consensus_queue.cc:260] T a63ea7226d914bbcafb98cba5585cd98 P 6117a0cd17234332b505b72d4001e76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66275db21bf14f86bec69e88109441a3" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 34931 } } peers { permanent_uuid: "6117a0cd17234332b505b72d4001e76f" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 41789 } } peers { permanent_uuid: "ca31005266f8418db3a0a363707bf6ad" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43219 } }
I20260810 23:30:26.756482 12656 ts_tablet_manager.cc:1428] T a63ea7226d914bbcafb98cba5585cd98 P 6117a0cd17234332b505b72d4001e76f: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260810 23:30:26.757686 12412 heartbeater.cc:503] Master 127.14.175.126:38699 was elected leader, sending a full tablet report...
I20260810 23:30:26.758291 12655 ts_tablet_manager.cc:1428] T a63ea7226d914bbcafb98cba5585cd98 P 66275db21bf14f86bec69e88109441a3: Time spent starting tablet: real 0.019s user 0.018s sys 0.000s
W20260810 23:30:26.796459 12415 tablet.cc:2369] T a63ea7226d914bbcafb98cba5585cd98 P 66275db21bf14f86bec69e88109441a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:30:26.857225 12626 tablet.cc:2369] T a63ea7226d914bbcafb98cba5585cd98 P ca31005266f8418db3a0a363707bf6ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:30:26.969861 12662 raft_consensus.cc:482] T a63ea7226d914bbcafb98cba5585cd98 P ca31005266f8418db3a0a363707bf6ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:30:26.970218 12662 raft_consensus.cc:504] T a63ea7226d914bbcafb98cba5585cd98 P ca31005266f8418db3a0a363707bf6ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66275db21bf14f86bec69e88109441a3" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 34931 } } peers { permanent_uuid: "6117a0cd17234332b505b72d4001e76f" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 41789 } } peers { permanent_uuid: "ca31005266f8418db3a0a363707bf6ad" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43219 } }
I20260810 23:30:26.971999 12662 leader_election.cc:288] T a63ea7226d914bbcafb98cba5585cd98 P ca31005266f8418db3a0a363707bf6ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66275db21bf14f86bec69e88109441a3 (127.14.175.66:34931), 6117a0cd17234332b505b72d4001e76f (127.14.175.65:41789)
I20260810 23:30:26.978085 12629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:26.972402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:34931 (local address 127.14.175.67:44539)
0810 23:30:26.972611 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:26.972632 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:26.972762 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:26.973129 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:26.973134 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:26.973203 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:30:26.973425 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:26.973441 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:26.974668 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:26.974672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:26.976369 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:26.976385 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:26.977544 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:26.977548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:26.977689 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:26.977715 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 23:30:26.977787 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 23:30:26.977883 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 23:30:26.978248 12647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:26.972511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:44539 (local address 127.14.175.66:34931)
0810 23:30:26.972703 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:26.972708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:26.972728 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:30:26.972909 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:26.972914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:26.972990 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:26.973083 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:30:26.973607 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:26.974535 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:26.976546 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:26.977411 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:26.977571 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:26.977647 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:26.977837 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 23:30:26.977970 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 23:30:26.979275 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:26.973067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:48051 (local address 127.14.175.65:41789)
0810 23:30:26.973333 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:26.973339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:26.973814 (+ 475us) server_negotiation.cc:407] Connection header received
0810 23:30:26.973943 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:26.973948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:26.974026 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:26.974114 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:30:26.974586 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:26.975363 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:26.977364 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:26.978509 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:26.978692 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:26.978765 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:26.979004 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 23:30:26.979099 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260810 23:30:26.979431 12666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:26.972881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:41789 (local address 127.14.175.67:48051)
0810 23:30:26.973711 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:26.973726 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:26.973821 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:26.974140 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:26.974143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:26.974160 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:30:26.974439 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:26.974447 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:26.975532 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:26.975537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:26.977192 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:26.977208 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:26.978688 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:26.978692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:26.978838 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:26.978870 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 23:30:26.978964 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 23:30:26.979089 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":485,"threads_started":1}
I20260810 23:30:26.979312 12337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63ea7226d914bbcafb98cba5585cd98" candidate_uuid: "ca31005266f8418db3a0a363707bf6ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66275db21bf14f86bec69e88109441a3" is_pre_election: true
I20260810 23:30:26.979954 12337 raft_consensus.cc:2454] T a63ea7226d914bbcafb98cba5585cd98 P 66275db21bf14f86bec69e88109441a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca31005266f8418db3a0a363707bf6ad in term 0.
I20260810 23:30:26.980423 12151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63ea7226d914bbcafb98cba5585cd98" candidate_uuid: "ca31005266f8418db3a0a363707bf6ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6117a0cd17234332b505b72d4001e76f" is_pre_election: true
I20260810 23:30:26.980711 12452 leader_election.cc:302] T a63ea7226d914bbcafb98cba5585cd98 P ca31005266f8418db3a0a363707bf6ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66275db21bf14f86bec69e88109441a3, ca31005266f8418db3a0a363707bf6ad; no voters:
I20260810 23:30:26.980976 12151 raft_consensus.cc:2454] T a63ea7226d914bbcafb98cba5585cd98 P 6117a0cd17234332b505b72d4001e76f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca31005266f8418db3a0a363707bf6ad in term 0.
I20260810 23:30:26.981103 12662 raft_consensus.cc:2781] T a63ea7226d914bbcafb98cba5585cd98 P ca31005266f8418db3a0a363707bf6ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:30:26.981281 12662 raft_consensus.cc:482] T a63ea7226d914bbcafb98cba5585cd98 P ca31005266f8418db3a0a363707bf6ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:30:26.981384 12662 raft_consensus.cc:3037] T a63ea7226d914bbcafb98cba5585cd98 P ca31005266f8418db3a0a363707bf6ad [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:30:26.983659 12662 raft_consensus.cc:504] T a63ea7226d914bbcafb98cba5585cd98 P ca31005266f8418db3a0a363707bf6ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66275db21bf14f86bec69e88109441a3" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 34931 } } peers { permanent_uuid: "6117a0cd17234332b505b72d4001e76f" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 41789 } } peers { permanent_uuid: "ca31005266f8418db3a0a363707bf6ad" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43219 } }
I20260810 23:30:26.984331 12662 leader_election.cc:288] T a63ea7226d914bbcafb98cba5585cd98 P ca31005266f8418db3a0a363707bf6ad [CANDIDATE]: Term 1 election: Requested vote from peers 66275db21bf14f86bec69e88109441a3 (127.14.175.66:34931), 6117a0cd17234332b505b72d4001e76f (127.14.175.65:41789)
I20260810 23:30:26.984771 12337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63ea7226d914bbcafb98cba5585cd98" candidate_uuid: "ca31005266f8418db3a0a363707bf6ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66275db21bf14f86bec69e88109441a3"
I20260810 23:30:26.984887 12151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63ea7226d914bbcafb98cba5585cd98" candidate_uuid: "ca31005266f8418db3a0a363707bf6ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6117a0cd17234332b505b72d4001e76f"
I20260810 23:30:26.984984 12337 raft_consensus.cc:3037] T a63ea7226d914bbcafb98cba5585cd98 P 66275db21bf14f86bec69e88109441a3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:30:26.985097 12151 raft_consensus.cc:3037] T a63ea7226d914bbcafb98cba5585cd98 P 6117a0cd17234332b505b72d4001e76f [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:30:26.987306 12337 raft_consensus.cc:2454] T a63ea7226d914bbcafb98cba5585cd98 P 66275db21bf14f86bec69e88109441a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca31005266f8418db3a0a363707bf6ad in term 1.
I20260810 23:30:26.987478 12151 raft_consensus.cc:2454] T a63ea7226d914bbcafb98cba5585cd98 P 6117a0cd17234332b505b72d4001e76f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca31005266f8418db3a0a363707bf6ad in term 1.
I20260810 23:30:26.987713 12452 leader_election.cc:302] T a63ea7226d914bbcafb98cba5585cd98 P ca31005266f8418db3a0a363707bf6ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66275db21bf14f86bec69e88109441a3, ca31005266f8418db3a0a363707bf6ad; no voters:
I20260810 23:30:26.988004 12662 raft_consensus.cc:2781] T a63ea7226d914bbcafb98cba5585cd98 P ca31005266f8418db3a0a363707bf6ad [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:30:26.988471 12662 raft_consensus.cc:686] T a63ea7226d914bbcafb98cba5585cd98 P ca31005266f8418db3a0a363707bf6ad [term 1 LEADER]: Becoming Leader. State: Replica: ca31005266f8418db3a0a363707bf6ad, State: Running, Role: LEADER
I20260810 23:30:26.988998 12662 consensus_queue.cc:237] T a63ea7226d914bbcafb98cba5585cd98 P ca31005266f8418db3a0a363707bf6ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66275db21bf14f86bec69e88109441a3" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 34931 } } peers { permanent_uuid: "6117a0cd17234332b505b72d4001e76f" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 41789 } } peers { permanent_uuid: "ca31005266f8418db3a0a363707bf6ad" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43219 } }
I20260810 23:30:26.993753 11937 catalog_manager.cc:5495] T a63ea7226d914bbcafb98cba5585cd98 P ca31005266f8418db3a0a363707bf6ad reported cstate change: term changed from 0 to 1, leader changed from <none> to ca31005266f8418db3a0a363707bf6ad (127.14.175.67). New cstate: current_term: 1 leader_uuid: "ca31005266f8418db3a0a363707bf6ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66275db21bf14f86bec69e88109441a3" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 34931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6117a0cd17234332b505b72d4001e76f" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 41789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca31005266f8418db3a0a363707bf6ad" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 43219 } health_report { overall_health: HEALTHY } } }
W20260810 23:30:27.005190 12218 tablet.cc:2369] T a63ea7226d914bbcafb98cba5585cd98 P 6117a0cd17234332b505b72d4001e76f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:30:27.069969 12649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:27.064048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51384 (local address 127.14.175.67:43219)
0810 23:30:27.064272 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:27.064279 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:27.064355 (+ 76us) server_negotiation.cc:407] Connection header received
0810 23:30:27.064579 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:27.064583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:27.064669 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:27.064808 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:30:27.065178 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:27.065927 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:27.067436 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:27.067895 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:27.067997 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:27.068756 (+ 759us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:30:27.069373 (+ 617us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:30:27.069520 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:27.069635 (+ 115us) server_negotiation.cc:299] Negotiation successful
0810 23:30:27.069682 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 23:30:27.078111 12151 raft_consensus.cc:1261] T a63ea7226d914bbcafb98cba5585cd98 P 6117a0cd17234332b505b72d4001e76f [term 1 FOLLOWER]: Refusing update from remote peer ca31005266f8418db3a0a363707bf6ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:30:27.078111 12337 raft_consensus.cc:1261] T a63ea7226d914bbcafb98cba5585cd98 P 66275db21bf14f86bec69e88109441a3 [term 1 FOLLOWER]: Refusing update from remote peer ca31005266f8418db3a0a363707bf6ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:30:27.079087 12669 consensus_queue.cc:1035] T a63ea7226d914bbcafb98cba5585cd98 P ca31005266f8418db3a0a363707bf6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6117a0cd17234332b505b72d4001e76f" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 41789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:30:27.079448 12662 consensus_queue.cc:1035] T a63ea7226d914bbcafb98cba5585cd98 P ca31005266f8418db3a0a363707bf6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "66275db21bf14f86bec69e88109441a3" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 34931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:30:27.090968 12673 mvcc.cc:205] Tried to move back new op lower bound from 7317113352501579776 to 7317113352151216128. Current Snapshot: MvccSnapshot[applied={T|T < 7317113352501579776}]
I20260810 23:30:27.104147 11933 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:30:27.104558 11933 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:30:27.109091 11933 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6d389901e914a7fab754036a0e860b8: Sending DeleteTablet for 3 replicas of tablet a63ea7226d914bbcafb98cba5585cd98
I20260810 23:30:27.110621 12316 tablet_service.cc:1514] Processing DeleteTablet for tablet a63ea7226d914bbcafb98cba5585cd98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:30:27 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:50686
I20260810 23:30:27.111603 12524 tablet_service.cc:1514] Processing DeleteTablet for tablet a63ea7226d914bbcafb98cba5585cd98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:30:27 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:51374
I20260810 23:30:27.111769 12690 tablet_replica.cc:330] T a63ea7226d914bbcafb98cba5585cd98 P 66275db21bf14f86bec69e88109441a3: stopping tablet replica
I20260810 23:30:27.112241 12690 raft_consensus.cc:2229] T a63ea7226d914bbcafb98cba5585cd98 P 66275db21bf14f86bec69e88109441a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:30:27.112355 12123 tablet_service.cc:1514] Processing DeleteTablet for tablet a63ea7226d914bbcafb98cba5585cd98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:30:27 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:56164
I20260810 23:30:27.112779 12690 raft_consensus.cc:2258] T a63ea7226d914bbcafb98cba5585cd98 P 66275db21bf14f86bec69e88109441a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:30:27.113006 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 12031
I20260810 23:30:27.118810 12690 ts_tablet_manager.cc:1905] T a63ea7226d914bbcafb98cba5585cd98 P 66275db21bf14f86bec69e88109441a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:30:27.125808 12693 tablet_replica.cc:330] T a63ea7226d914bbcafb98cba5585cd98 P 6117a0cd17234332b505b72d4001e76f: stopping tablet replica
I20260810 23:30:27.126423 12693 raft_consensus.cc:2229] T a63ea7226d914bbcafb98cba5585cd98 P 6117a0cd17234332b505b72d4001e76f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:30:27.127069 12693 raft_consensus.cc:2258] T a63ea7226d914bbcafb98cba5585cd98 P 6117a0cd17234332b505b72d4001e76f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:30:27.129968 12690 ts_tablet_manager.cc:1918] T a63ea7226d914bbcafb98cba5585cd98 P 66275db21bf14f86bec69e88109441a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:30:27.130180 12690 log.cc:1196] T a63ea7226d914bbcafb98cba5585cd98 P 66275db21bf14f86bec69e88109441a3: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-1/wal/wals/a63ea7226d914bbcafb98cba5585cd98
I20260810 23:30:27.130431 12693 ts_tablet_manager.cc:1905] T a63ea7226d914bbcafb98cba5585cd98 P 6117a0cd17234332b505b72d4001e76f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:30:27.130836 12690 ts_tablet_manager.cc:1939] T a63ea7226d914bbcafb98cba5585cd98 P 66275db21bf14f86bec69e88109441a3: Deleting consensus metadata
I20260810 23:30:27.132920 11916 catalog_manager.cc:4841] TS 66275db21bf14f86bec69e88109441a3 (127.14.175.66:34931): tablet a63ea7226d914bbcafb98cba5585cd98 (table test-table [id=d28e57600ab04346b79e21ba784539c8]) successfully deleted
I20260810 23:30:27.133237 12691 tablet_replica.cc:330] T a63ea7226d914bbcafb98cba5585cd98 P ca31005266f8418db3a0a363707bf6ad: stopping tablet replica
I20260810 23:30:27.133711 12691 raft_consensus.cc:2229] T a63ea7226d914bbcafb98cba5585cd98 P ca31005266f8418db3a0a363707bf6ad [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:30:27.134413 12691 raft_consensus.cc:2258] T a63ea7226d914bbcafb98cba5585cd98 P ca31005266f8418db3a0a363707bf6ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:30:27.136762 12691 ts_tablet_manager.cc:1905] T a63ea7226d914bbcafb98cba5585cd98 P ca31005266f8418db3a0a363707bf6ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:30:27.138094 12693 ts_tablet_manager.cc:1918] T a63ea7226d914bbcafb98cba5585cd98 P 6117a0cd17234332b505b72d4001e76f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:30:27.138291 12693 log.cc:1196] T a63ea7226d914bbcafb98cba5585cd98 P 6117a0cd17234332b505b72d4001e76f: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-0/wal/wals/a63ea7226d914bbcafb98cba5585cd98
I20260810 23:30:27.138836 12693 ts_tablet_manager.cc:1939] T a63ea7226d914bbcafb98cba5585cd98 P 6117a0cd17234332b505b72d4001e76f: Deleting consensus metadata
I20260810 23:30:27.140825 11916 catalog_manager.cc:4841] TS 6117a0cd17234332b505b72d4001e76f (127.14.175.65:41789): tablet a63ea7226d914bbcafb98cba5585cd98 (table test-table [id=d28e57600ab04346b79e21ba784539c8]) successfully deleted
Aug 10 23:30:27 dist-test-slave-rmhg krb5kdc[11862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.65@KRBTEST.COM
I20260810 23:30:27.144425 12691 ts_tablet_manager.cc:1918] T a63ea7226d914bbcafb98cba5585cd98 P ca31005266f8418db3a0a363707bf6ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:30:27.144677 12691 log.cc:1196] T a63ea7226d914bbcafb98cba5585cd98 P ca31005266f8418db3a0a363707bf6ad: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404299514076-15037-0/minicluster-data/ts-2/wal/wals/a63ea7226d914bbcafb98cba5585cd98
I20260810 23:30:27.145304 12691 ts_tablet_manager.cc:1939] T a63ea7226d914bbcafb98cba5585cd98 P ca31005266f8418db3a0a363707bf6ad: Deleting consensus metadata
I20260810 23:30:27.147408 11912 catalog_manager.cc:4841] TS ca31005266f8418db3a0a363707bf6ad (127.14.175.67:43219): tablet a63ea7226d914bbcafb98cba5585cd98 (table test-table [id=d28e57600ab04346b79e21ba784539c8]) successfully deleted
I20260810 23:30:27.150801 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:27.114101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56174 (local address 127.14.175.65:41789)
0810 23:30:27.114290 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:27.114297 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:27.127999 (+ 13702us) server_negotiation.cc:407] Connection header received
0810 23:30:27.128226 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:27.128232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:27.128340 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:27.128459 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:27.130832 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:27.132042 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:27.135763 (+ 3721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:27.136488 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:27.136640 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:27.142928 (+ 6288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:27.142946 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:27.142968 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:30:27.143020 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:27.147273 (+ 4253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:27.148255 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:27.148268 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:27.148272 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:27.148473 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:27.148869 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:27.148875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:27.148878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:27.149466 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:27.149820 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:27.150148 (+ 328us) server_negotiation.cc:299] Negotiation successful
0810 23:30:27.150410 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 23:30:27.151561 12208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:30:27.243877 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 12031
I20260810 23:30:27.270512 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 12226
Aug 10 23:30:27 dist-test-slave-rmhg krb5kdc[11862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.66@KRBTEST.COM
I20260810 23:30:27.286723 12647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:27.271354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50700 (local address 127.14.175.66:34931)
0810 23:30:27.271535 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:27.271541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:27.271629 (+ 88us) server_negotiation.cc:407] Connection header received
0810 23:30:27.271865 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:27.271871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:27.271949 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:27.272037 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:27.273343 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:27.274069 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:27.275361 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:27.275778 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:27.275891 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:27.280515 (+ 4624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:27.280532 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:27.280549 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:30:27.280588 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:27.283551 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:27.284383 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:27.284391 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:27.284394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:27.284571 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:27.284970 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:27.284975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:27.284977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:27.285470 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:27.285797 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:27.286129 (+ 332us) server_negotiation.cc:299] Negotiation successful
0810 23:30:27.286368 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 23:30:27.287343 12404 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:30:27.434388 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 12226
I20260810 23:30:27.460852 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 12425
Aug 10 23:30:27 dist-test-slave-rmhg krb5kdc[11862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.67@KRBTEST.COM
I20260810 23:30:27.478120 12649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:27.461749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51390 (local address 127.14.175.67:43219)
0810 23:30:27.461942 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:27.461950 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:27.461972 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:30:27.462054 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:27.462058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:27.462122 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:27.462216 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:27.463703 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:27.464590 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:27.466108 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:27.466585 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:27.466707 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:27.471778 (+ 5071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:27.471794 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:27.471811 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:30:27.471857 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:27.475050 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:27.475805 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:27.475814 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:27.475816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:27.476006 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:27.476403 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:27.476406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:27.476409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:27.476829 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:27.477145 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:27.477502 (+ 357us) server_negotiation.cc:299] Negotiation successful
0810 23:30:27.477742 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 23:30:27.479034 12614 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:30:27.605054 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 12425
I20260810 23:30:27.630242 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 11886
I20260810 23:30:27.631381 11976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:30:27.743851 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 11886
2026-08-10T23:30:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4970 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T23:30:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:30:27Z Disabled control of system clock
I20260810 23:30:27.843730 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:43221
--webserver_interface=127.14.175.126
--webserver_port=0
--builtin_ntp_servers=127.14.175.84:46753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:43221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:30:28.196255 12709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:30:28.196561 12709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:30:28.196699 12709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:30:28.204068 12709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:30:28.204210 12709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:30:28.204279 12709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:30:28.204360 12709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:30:28.204419 12709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:30:28.214458 12709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:46753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:43221
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:43221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:30:28.215915 12709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:30:28.217926 12709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:30:28.230610 12720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:28.230701 12718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:28.230581 12722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:30:28.231432 12709 server_base.cc:1034] running on GCE node
I20260810 23:30:28.232327 12709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:30:28.233928 12709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:30:28.235136 12709 hybrid_clock.cc:648] HybridClock initialized: now 1786404628235090 us; error 39 us; skew 500 ppm
I20260810 23:30:28.235505 12709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:30:28.237933 12709 webserver.cc:460] Webserver started at http://127.14.175.126:40011/ using document root <none> and password file <none>
I20260810 23:30:28.238631 12709 fs_manager.cc:359] Metadata directory not provided
I20260810 23:30:28.238739 12709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:30:28.239089 12709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:30:28.242183 12709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "80403775157b4f28817d61569ef3cfd0"
format_stamp: "Formatted at 2026-08-10 23:30:28 on dist-test-slave-rmhg"
server_key: "01229e7d2c0858601940a4b17bb74d4e"
server_key_iv: "c01ef69611edd46f181b96bc143e2bb8"
server_key_version: "encryptionkey@0"
I20260810 23:30:28.243021 12709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "80403775157b4f28817d61569ef3cfd0"
format_stamp: "Formatted at 2026-08-10 23:30:28 on dist-test-slave-rmhg"
server_key: "01229e7d2c0858601940a4b17bb74d4e"
server_key_iv: "c01ef69611edd46f181b96bc143e2bb8"
server_key_version: "encryptionkey@0"
I20260810 23:30:28.248476 12709 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:30:28.252379 12730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:28.253985 12709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:30:28.254173 12709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "80403775157b4f28817d61569ef3cfd0"
format_stamp: "Formatted at 2026-08-10 23:30:28 on dist-test-slave-rmhg"
server_key: "01229e7d2c0858601940a4b17bb74d4e"
server_key_iv: "c01ef69611edd46f181b96bc143e2bb8"
server_key_version: "encryptionkey@0"
I20260810 23:30:28.254395 12709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:30:28.279628 12709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:30:28.280565 12709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:30:28.280879 12709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:30:28.300110 12709 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:43221
I20260810 23:30:28.300130 12806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:43221 every 8 connection(s)
I20260810 23:30:28.301630 12709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:30:28.305857 12810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:30:28.306440 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 12709
I20260810 23:30:28.306640 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:30:28.307265 15037 external_mini_cluster.cc:1428] Setting key 2b08b4570622724a336a8e9b519d6764
I20260810 23:30:28.319518 12810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80403775157b4f28817d61569ef3cfd0: Bootstrap starting.
I20260810 23:30:28.322068 12815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:28.309530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52312 (local address 127.14.175.126:43221)
0810 23:30:28.310700 (+ 1170us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:28.310713 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:28.310760 (+ 47us) server_negotiation.cc:407] Connection header received
0810 23:30:28.312803 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:28.312833 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:28.313448 (+ 615us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:28.313876 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:28.314533 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:28.315899 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:28.317417 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:28.318171 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:28.318360 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:28.319559 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:28.319568 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:28.319584 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:28.320219 (+ 635us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:28.320308 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:28.320331 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:28.320434 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:28.320689 (+ 255us) server_negotiation.cc:299] Negotiation successful
0810 23:30:28.321021 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":417,"threads_started":1}
I20260810 23:30:28.323110 12810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80403775157b4f28817d61569ef3cfd0: Neither blocks nor log segments found. Creating new log.
I20260810 23:30:28.324548 12810 log.cc:824] T 00000000000000000000000000000000 P 80403775157b4f28817d61569ef3cfd0: Log is configured to *not* fsync() on all Append() calls
I20260810 23:30:28.327579 12810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80403775157b4f28817d61569ef3cfd0: No bootstrap required, opened a new log
I20260810 23:30:28.335621 12810 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80403775157b4f28817d61569ef3cfd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80403775157b4f28817d61569ef3cfd0" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43221 } }
I20260810 23:30:28.336169 12810 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80403775157b4f28817d61569ef3cfd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:30:28.336308 12810 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80403775157b4f28817d61569ef3cfd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80403775157b4f28817d61569ef3cfd0, State: Initialized, Role: FOLLOWER
I20260810 23:30:28.337018 12810 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80403775157b4f28817d61569ef3cfd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80403775157b4f28817d61569ef3cfd0" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43221 } }
I20260810 23:30:28.337319 12810 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80403775157b4f28817d61569ef3cfd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:30:28.337466 12810 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80403775157b4f28817d61569ef3cfd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:30:28.337623 12810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80403775157b4f28817d61569ef3cfd0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:30:28.339943 12810 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80403775157b4f28817d61569ef3cfd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80403775157b4f28817d61569ef3cfd0" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43221 } }
I20260810 23:30:28.340545 12810 leader_election.cc:302] T 00000000000000000000000000000000 P 80403775157b4f28817d61569ef3cfd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80403775157b4f28817d61569ef3cfd0; no voters:
I20260810 23:30:28.341253 12810 leader_election.cc:288] T 00000000000000000000000000000000 P 80403775157b4f28817d61569ef3cfd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:30:28.341409 12823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80403775157b4f28817d61569ef3cfd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:30:28.342216 12823 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80403775157b4f28817d61569ef3cfd0 [term 1 LEADER]: Becoming Leader. State: Replica: 80403775157b4f28817d61569ef3cfd0, State: Running, Role: LEADER
I20260810 23:30:28.342829 12823 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80403775157b4f28817d61569ef3cfd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80403775157b4f28817d61569ef3cfd0" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43221 } }
I20260810 23:30:28.343643 12810 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80403775157b4f28817d61569ef3cfd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:30:28.346115 12824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80403775157b4f28817d61569ef3cfd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80403775157b4f28817d61569ef3cfd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80403775157b4f28817d61569ef3cfd0" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43221 } } }
I20260810 23:30:28.346390 12825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80403775157b4f28817d61569ef3cfd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80403775157b4f28817d61569ef3cfd0. Latest consensus state: current_term: 1 leader_uuid: "80403775157b4f28817d61569ef3cfd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80403775157b4f28817d61569ef3cfd0" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43221 } } }
I20260810 23:30:28.346618 12824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80403775157b4f28817d61569ef3cfd0 [sys.catalog]: This master's current role is: LEADER
I20260810 23:30:28.346704 12825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80403775157b4f28817d61569ef3cfd0 [sys.catalog]: This master's current role is: LEADER
I20260810 23:30:28.348625 12838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:30:28.353691 12838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:30:28.367599 12838 catalog_manager.cc:1347] Generated new cluster ID: e075be6aaaaa49a7a94f208db773b4b2
I20260810 23:30:28.367753 12838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:30:28.401002 12838 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:30:28.402379 12838 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:30:28.413414 12838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80403775157b4f28817d61569ef3cfd0: Generated new TSK 0
I20260810 23:30:28.414353 12838 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:30:28.426991 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.65:0
--local_ip_for_outbound_sockets=127.14.175.65
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:43221
--builtin_ntp_servers=127.14.175.84:46753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:30:28.787300 12856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:30:28.787603 12856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:30:28.787729 12856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:30:28.795351 12856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:30:28.795480 12856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:30:28.795636 12856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.65
I20260810 23:30:28.805342 12856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:46753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:43221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.65
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:30:28.806802 12856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:30:28.808790 12856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:30:28.822731 12865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:28.822685 12866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:28.822760 12868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:30:28.824100 12856 server_base.cc:1034] running on GCE node
I20260810 23:30:28.824782 12856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:30:28.825910 12856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:30:28.827176 12856 hybrid_clock.cc:648] HybridClock initialized: now 1786404628827101 us; error 65 us; skew 500 ppm
I20260810 23:30:28.827535 12856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:30:28.830050 12856 webserver.cc:460] Webserver started at http://127.14.175.65:33971/ using document root <none> and password file <none>
I20260810 23:30:28.830792 12856 fs_manager.cc:359] Metadata directory not provided
I20260810 23:30:28.830904 12856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:30:28.831233 12856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:30:28.834306 12856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-0/data/instance:
uuid: "ea53dd0dce004ac6b5e80f10f9f463ac"
format_stamp: "Formatted at 2026-08-10 23:30:28 on dist-test-slave-rmhg"
server_key: "c645e4cd0b0a09eadea972432ef6fa97"
server_key_iv: "7a84417f2b4c718bcb6d691ed784c5f6"
server_key_version: "encryptionkey@0"
I20260810 23:30:28.835126 12856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance:
uuid: "ea53dd0dce004ac6b5e80f10f9f463ac"
format_stamp: "Formatted at 2026-08-10 23:30:28 on dist-test-slave-rmhg"
server_key: "c645e4cd0b0a09eadea972432ef6fa97"
server_key_iv: "7a84417f2b4c718bcb6d691ed784c5f6"
server_key_version: "encryptionkey@0"
I20260810 23:30:28.840513 12856 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:30:28.844542 12876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:28.846143 12856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:30:28.846311 12856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-0/wal
uuid: "ea53dd0dce004ac6b5e80f10f9f463ac"
format_stamp: "Formatted at 2026-08-10 23:30:28 on dist-test-slave-rmhg"
server_key: "c645e4cd0b0a09eadea972432ef6fa97"
server_key_iv: "7a84417f2b4c718bcb6d691ed784c5f6"
server_key_version: "encryptionkey@0"
I20260810 23:30:28.846516 12856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:30:28.873237 12856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:30:28.874171 12856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:30:28.874461 12856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:30:28.875523 12856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:30:28.877106 12856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:30:28.877233 12856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260810 23:30:28.877329 12856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:30:28.877393 12856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:28.909188 12856 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.65:45691
I20260810 23:30:28.909273 13052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.65:45691 every 8 connection(s)
I20260810 23:30:28.910570 12856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
I20260810 23:30:28.911103 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 12856
I20260810 23:30:28.911283 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance
I20260810 23:30:28.911868 15037 external_mini_cluster.cc:1428] Setting key ec6fcee7212023c0f483586904dcd0bd
I20260810 23:30:28.915373 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.66:0
--local_ip_for_outbound_sockets=127.14.175.66
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:43221
--builtin_ntp_servers=127.14.175.84:46753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:30:28.925789 13058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:28.913560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:50295 (local address 127.14.175.126:43221)
0810 23:30:28.914025 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:28.914032 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:28.915773 (+ 1741us) server_negotiation.cc:407] Connection header received
0810 23:30:28.918291 (+ 2518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:28.918296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:28.918367 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:28.918496 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:28.919920 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:28.920676 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:28.922843 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:28.923275 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:28.923390 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:28.925016 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:28.925021 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:28.925025 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:28.925045 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:28.925088 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:28.925101 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:28.925206 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:28.925421 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 23:30:28.925511 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":223,"threads_started":1}
I20260810 23:30:28.927222 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:28.914399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43221 (local address 127.14.175.65:50295)
0810 23:30:28.915561 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:28.915640 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:28.917943 (+ 2303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:28.918746 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:28.918758 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:28.918893 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:30:28.919711 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:28.919735 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:28.920828 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:28.920833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:28.922696 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:28.922711 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:28.923558 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:28.923565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:28.923860 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:28.924617 (+ 757us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:30:28.924620 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:30:28.924686 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:30:28.924799 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:30:28.925207 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:30:28.925218 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:30:28.925226 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 23:30:28.925416 (+ 190us) client_negotiation.cc:240] Negotiation successful
0810 23:30:28.925851 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":309,"threads_started":1}
I20260810 23:30:28.928953 13053 heartbeater.cc:347] Connected to a master server at 127.14.175.126:43221
I20260810 23:30:28.929313 13053 heartbeater.cc:464] Registering TS with master...
I20260810 23:30:28.930102 13053 heartbeater.cc:511] Master 127.14.175.126:43221 requested a full tablet report, sending...
I20260810 23:30:28.932544 12755 ts_manager.cc:194] Registered new tserver with Master: ea53dd0dce004ac6b5e80f10f9f463ac (127.14.175.65:45691)
I20260810 23:30:28.934542 12755 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.175.65:50295
W20260810 23:30:29.268553 13061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:30:29.268852 13061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:30:29.268939 13061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:30:29.276445 13061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:30:29.276553 13061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:30:29.276659 13061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.66
I20260810 23:30:29.286018 13061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:46753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:43221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.66
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:30:29.287298 13061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:30:29.289080 13061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:30:29.302984 13069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:29.303586 13071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:29.303582 13068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:30:29.304014 13061 server_base.cc:1034] running on GCE node
I20260810 23:30:29.304699 13061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:30:29.305899 13061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:30:29.307188 13061 hybrid_clock.cc:648] HybridClock initialized: now 1786404629307135 us; error 76 us; skew 500 ppm
I20260810 23:30:29.307528 13061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:30:29.310011 13061 webserver.cc:460] Webserver started at http://127.14.175.66:46763/ using document root <none> and password file <none>
I20260810 23:30:29.310721 13061 fs_manager.cc:359] Metadata directory not provided
I20260810 23:30:29.310824 13061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:30:29.311151 13061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:30:29.314174 13061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-1/data/instance:
uuid: "c8f21a75c82642e19547c25ac88bab2b"
format_stamp: "Formatted at 2026-08-10 23:30:29 on dist-test-slave-rmhg"
server_key: "46942bbac2e4da3ecb6f0b453a638bde"
server_key_iv: "aa0e0eedf921c7e3c930eb36edfddec2"
server_key_version: "encryptionkey@0"
I20260810 23:30:29.314972 13061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance:
uuid: "c8f21a75c82642e19547c25ac88bab2b"
format_stamp: "Formatted at 2026-08-10 23:30:29 on dist-test-slave-rmhg"
server_key: "46942bbac2e4da3ecb6f0b453a638bde"
server_key_iv: "aa0e0eedf921c7e3c930eb36edfddec2"
server_key_version: "encryptionkey@0"
I20260810 23:30:29.320489 13061 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:30:29.324361 13079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:29.325878 13061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:30:29.326040 13061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-1/wal
uuid: "c8f21a75c82642e19547c25ac88bab2b"
format_stamp: "Formatted at 2026-08-10 23:30:29 on dist-test-slave-rmhg"
server_key: "46942bbac2e4da3ecb6f0b453a638bde"
server_key_iv: "aa0e0eedf921c7e3c930eb36edfddec2"
server_key_version: "encryptionkey@0"
I20260810 23:30:29.326256 13061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:30:29.342936 13061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:30:29.343869 13061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:30:29.344165 13061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:30:29.345286 13061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:30:29.346810 13061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:30:29.346911 13061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:29.347014 13061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:30:29.347082 13061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:29.378372 13061 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.66:42477
I20260810 23:30:29.378424 13259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.66:42477 every 8 connection(s)
I20260810 23:30:29.379772 13061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
I20260810 23:30:29.381538 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 13061
I20260810 23:30:29.381707 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance
I20260810 23:30:29.382290 15037 external_mini_cluster.cc:1428] Setting key 6cbe0190e8cef014e145216f1049a1f4
I20260810 23:30:29.385288 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.67:0
--local_ip_for_outbound_sockets=127.14.175.67
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:43221
--builtin_ntp_servers=127.14.175.84:46753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:30:29.393934 13058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:29.382575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:48601 (local address 127.14.175.126:43221)
0810 23:30:29.382773 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:29.382778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:29.384577 (+ 1799us) server_negotiation.cc:407] Connection header received
0810 23:30:29.386534 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:29.386541 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:29.386637 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:29.386742 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:29.387985 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:29.388938 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:29.390942 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:29.391492 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:29.391614 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:29.393101 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:29.393107 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:29.393111 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:29.393135 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:29.393203 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:29.393220 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:29.393304 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:29.393516 (+ 212us) server_negotiation.cc:299] Negotiation successful
0810 23:30:29.393622 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 23:30:29.395076 13263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:29.383250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43221 (local address 127.14.175.66:48601)
0810 23:30:29.384380 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:29.384438 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:29.386219 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:29.386980 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:29.386991 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:29.387102 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:30:29.387803 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:29.387824 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:29.389083 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:29.389088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:29.390785 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:29.390801 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:29.391717 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:29.391723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:29.391980 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:29.392742 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:30:29.392745 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:30:29.392797 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:30:29.392894 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:30:29.393307 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:30:29.393317 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:30:29.393325 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 23:30:29.393495 (+ 170us) client_negotiation.cc:240] Negotiation successful
0810 23:30:29.393910 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":374,"threads_started":1}
I20260810 23:30:29.396505 13261 heartbeater.cc:347] Connected to a master server at 127.14.175.126:43221
I20260810 23:30:29.396827 13261 heartbeater.cc:464] Registering TS with master...
I20260810 23:30:29.397738 13261 heartbeater.cc:511] Master 127.14.175.126:43221 requested a full tablet report, sending...
I20260810 23:30:29.399307 12755 ts_manager.cc:194] Registered new tserver with Master: c8f21a75c82642e19547c25ac88bab2b (127.14.175.66:42477)
I20260810 23:30:29.400091 12755 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.175.66:48601
W20260810 23:30:29.735954 13265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:30:29.736276 13265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:30:29.736398 13265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:30:29.744046 13265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:30:29.744176 13265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:30:29.744331 13265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.67
I20260810 23:30:29.754688 13265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:46753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:43221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.67
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:30:29.756145 13265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:30:29.758191 13265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:30:29.772593 13278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:29.772608 13281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:30:29.773238 13265 server_base.cc:1034] running on GCE node
W20260810 23:30:29.773286 13277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:30:29.774045 13265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:30:29.775135 13265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:30:29.776383 13265 hybrid_clock.cc:648] HybridClock initialized: now 1786404629776325 us; error 52 us; skew 500 ppm
I20260810 23:30:29.776751 13265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:30:29.779249 13265 webserver.cc:460] Webserver started at http://127.14.175.67:42163/ using document root <none> and password file <none>
I20260810 23:30:29.779995 13265 fs_manager.cc:359] Metadata directory not provided
I20260810 23:30:29.780122 13265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:30:29.780445 13265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:30:29.783607 13265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-2/data/instance:
uuid: "1f523a91c3604a1b8d1c2aced77f77ca"
format_stamp: "Formatted at 2026-08-10 23:30:29 on dist-test-slave-rmhg"
server_key: "64a826e576a1c5a10d8785dd990a67e5"
server_key_iv: "63c2067b855dbb0252cda2de967d5069"
server_key_version: "encryptionkey@0"
I20260810 23:30:29.784446 13265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance:
uuid: "1f523a91c3604a1b8d1c2aced77f77ca"
format_stamp: "Formatted at 2026-08-10 23:30:29 on dist-test-slave-rmhg"
server_key: "64a826e576a1c5a10d8785dd990a67e5"
server_key_iv: "63c2067b855dbb0252cda2de967d5069"
server_key_version: "encryptionkey@0"
I20260810 23:30:29.790330 13265 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260810 23:30:29.794169 13288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:29.795585 13265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260810 23:30:29.795768 13265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-2/wal
uuid: "1f523a91c3604a1b8d1c2aced77f77ca"
format_stamp: "Formatted at 2026-08-10 23:30:29 on dist-test-slave-rmhg"
server_key: "64a826e576a1c5a10d8785dd990a67e5"
server_key_iv: "63c2067b855dbb0252cda2de967d5069"
server_key_version: "encryptionkey@0"
I20260810 23:30:29.795982 13265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:30:29.822644 13265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:30:29.823566 13265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:30:29.823886 13265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:30:29.824960 13265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:30:29.826625 13265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:30:29.826738 13265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:29.826845 13265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:30:29.826917 13265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:29.858863 13265 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.67:44457
I20260810 23:30:29.858928 13458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.67:44457 every 8 connection(s)
I20260810 23:30:29.860339 13265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
I20260810 23:30:29.868185 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 13265
I20260810 23:30:29.868399 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance
I20260810 23:30:29.868925 15037 external_mini_cluster.cc:1428] Setting key 4e820ccf5c8bef8b27adaff7b3204dcf
I20260810 23:30:29.873447 13058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:29.862933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:34587 (local address 127.14.175.126:43221)
0810 23:30:29.863148 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:29.863154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:29.864734 (+ 1580us) server_negotiation.cc:407] Connection header received
0810 23:30:29.866623 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:29.866635 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:29.866709 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:29.866802 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:29.867907 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:29.868989 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:29.870715 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:29.871230 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:29.871327 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:29.872733 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:29.872738 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:29.872741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:29.872759 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:29.872795 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:29.872807 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:29.872877 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:29.873054 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 23:30:29.873141 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 23:30:29.874542 13463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:29.863568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:43221 (local address 127.14.175.67:34587)
0810 23:30:29.864539 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:29.864590 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:29.866337 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:29.867003 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:29.867012 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:29.867116 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:30:29.867738 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:29.867757 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:29.869142 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:29.869171 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:29.870576 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:29.870588 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:29.871461 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:29.871469 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:29.871756 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:29.872392 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:30:29.872394 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:30:29.872438 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:30:29.872532 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:30:29.872882 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:30:29.872889 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:30:29.872894 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:30:29.873040 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 23:30:29.873419 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":272,"threads_started":1}
I20260810 23:30:29.875996 13461 heartbeater.cc:347] Connected to a master server at 127.14.175.126:43221
I20260810 23:30:29.876327 13461 heartbeater.cc:464] Registering TS with master...
I20260810 23:30:29.877108 13461 heartbeater.cc:511] Master 127.14.175.126:43221 requested a full tablet report, sending...
I20260810 23:30:29.878634 12755 ts_manager.cc:194] Registered new tserver with Master: 1f523a91c3604a1b8d1c2aced77f77ca (127.14.175.67:44457)
I20260810 23:30:29.879441 12755 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.175.67:34587
I20260810 23:30:29.884109 15037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:30:29.894748 13058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:29.888408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56498 (local address 127.14.175.126:43221)
0810 23:30:29.888715 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:29.888722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:29.889012 (+ 290us) server_negotiation.cc:407] Connection header received
0810 23:30:29.889205 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:29.889210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:29.889290 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:29.889380 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:29.889968 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:29.890710 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:29.892146 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:29.892575 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:29.892693 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:29.893992 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:29.893997 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:29.894000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:29.894018 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:29.894051 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:29.894063 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:29.894220 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:29.894413 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 23:30:29.894475 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260810 23:30:29.899044 12755 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:30:29.906339 12755 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:30:29.934494 13482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:29.919780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:44457 (local address 127.0.0.1:43356)
0810 23:30:29.920741 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:29.920802 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:29.920996 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:29.923565 (+ 2569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:29.923571 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:29.923604 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:30:29.924076 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:29.924088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:29.926878 (+ 2790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:29.926885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:29.929808 (+ 2923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:29.929832 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:29.933190 (+ 3358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:29.933199 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:29.933466 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:29.933536 (+ 70us) client_negotiation.cc:769] Sending connection context
0810 23:30:29.933676 (+ 140us) client_negotiation.cc:240] Negotiation successful
0810 23:30:29.933929 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":1,"thread_start_us":1538,"threads_started":1}
I20260810 23:30:29.934563 13481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:29.919932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:42477 (local address 127.0.0.1:45934)
0810 23:30:29.921609 (+ 1677us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:29.921629 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:29.921773 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:29.923186 (+ 1413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:29.923194 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:29.923227 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:30:29.923785 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:29.923806 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:29.927802 (+ 3996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:29.927809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:29.930691 (+ 2882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:29.930708 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:29.933495 (+ 2787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:29.933503 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:29.933751 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:29.933791 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 23:30:29.933859 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 23:30:29.934022 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1548,"mutex_wait_us":363,"thread_start_us":304,"threads_started":1}
I20260810 23:30:29.936872 13483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:29.920041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43356 (local address 127.14.175.67:44457)
0810 23:30:29.922883 (+ 2842us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:29.922893 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:29.922934 (+ 41us) server_negotiation.cc:407] Connection header received
0810 23:30:29.923057 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:29.923065 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:29.923315 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:29.924320 (+ 1005us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:30:29.924656 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:29.926681 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:29.931038 (+ 4357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:29.932872 (+ 1834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:29.935758 (+ 2886us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:29.936168 (+ 410us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:29.936260 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 23:30:29.936479 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2654,"thread_start_us":2502,"threads_started":1}
I20260810 23:30:29.937544 13486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:29.920358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45934 (local address 127.14.175.66:42477)
0810 23:30:29.922446 (+ 2088us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:29.922457 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:29.922491 (+ 34us) server_negotiation.cc:407] Connection header received
0810 23:30:29.922610 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:29.922618 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:29.922906 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:29.925624 (+ 2718us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:30:29.925923 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:29.927597 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:29.931504 (+ 3907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:29.933220 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:29.936670 (+ 3450us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:29.936929 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:29.937016 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 23:30:29.937212 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1905,"thread_start_us":1801,"threads_started":1}
I20260810 23:30:29.939512 13053 heartbeater.cc:503] Master 127.14.175.126:43221 was elected leader, sending a full tablet report...
I20260810 23:30:29.942022 13363 tablet_service.cc:1467] Processing CreateTablet for tablet 56bf396f8336451e918cbac5f04d6446 (DEFAULT_TABLE table=test-table [id=d590b3103c4e47a38cc650a56e50db86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:30:29.943073 13163 tablet_service.cc:1467] Processing CreateTablet for tablet 56bf396f8336451e918cbac5f04d6446 (DEFAULT_TABLE table=test-table [id=d590b3103c4e47a38cc650a56e50db86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:30:29.943846 13363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56bf396f8336451e918cbac5f04d6446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:30:29.944378 13163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56bf396f8336451e918cbac5f04d6446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:30:29.949492 13482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:29.922714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:45691 (local address 127.0.0.1:58764)
0810 23:30:29.934834 (+ 12120us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:29.934858 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:29.934983 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:29.942438 (+ 7455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:29.942443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:29.942475 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:30:29.942672 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:29.942687 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:29.944744 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:29.944750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:29.946596 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:29.946617 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:29.948662 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:29.948668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:29.948863 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:29.948902 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 23:30:29.948972 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 23:30:29.949142 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11988,"thread_start_us":14600,"threads_started":1}
I20260810 23:30:29.950901 13488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:29.923251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58764 (local address 127.14.175.65:45691)
0810 23:30:29.940194 (+ 16943us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:29.940204 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:29.940239 (+ 35us) server_negotiation.cc:407] Connection header received
0810 23:30:29.940349 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:29.940358 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:29.942160 (+ 1802us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:29.942478 (+ 318us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:30:29.942865 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:29.944564 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:29.946863 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:29.948447 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:29.949895 (+ 1448us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:29.950215 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:29.950299 (+ 84us) server_negotiation.cc:299] Negotiation successful
0810 23:30:29.950524 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16735,"thread_start_us":18298,"threads_started":1}
I20260810 23:30:29.955466 12956 tablet_service.cc:1467] Processing CreateTablet for tablet 56bf396f8336451e918cbac5f04d6446 (DEFAULT_TABLE table=test-table [id=d590b3103c4e47a38cc650a56e50db86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:30:29.957000 13489 tablet_bootstrap.cc:492] T 56bf396f8336451e918cbac5f04d6446 P c8f21a75c82642e19547c25ac88bab2b: Bootstrap starting.
I20260810 23:30:29.957247 12956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56bf396f8336451e918cbac5f04d6446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:30:29.960292 13491 tablet_bootstrap.cc:492] T 56bf396f8336451e918cbac5f04d6446 P 1f523a91c3604a1b8d1c2aced77f77ca: Bootstrap starting.
I20260810 23:30:29.960624 13489 tablet_bootstrap.cc:654] T 56bf396f8336451e918cbac5f04d6446 P c8f21a75c82642e19547c25ac88bab2b: Neither blocks nor log segments found. Creating new log.
I20260810 23:30:29.962102 13489 log.cc:824] T 56bf396f8336451e918cbac5f04d6446 P c8f21a75c82642e19547c25ac88bab2b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:30:29.963999 13491 tablet_bootstrap.cc:654] T 56bf396f8336451e918cbac5f04d6446 P 1f523a91c3604a1b8d1c2aced77f77ca: Neither blocks nor log segments found. Creating new log.
I20260810 23:30:29.965498 13491 log.cc:824] T 56bf396f8336451e918cbac5f04d6446 P 1f523a91c3604a1b8d1c2aced77f77ca: Log is configured to *not* fsync() on all Append() calls
I20260810 23:30:29.966377 13489 tablet_bootstrap.cc:492] T 56bf396f8336451e918cbac5f04d6446 P c8f21a75c82642e19547c25ac88bab2b: No bootstrap required, opened a new log
I20260810 23:30:29.966699 13489 ts_tablet_manager.cc:1397] T 56bf396f8336451e918cbac5f04d6446 P c8f21a75c82642e19547c25ac88bab2b: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260810 23:30:29.968884 13491 tablet_bootstrap.cc:492] T 56bf396f8336451e918cbac5f04d6446 P 1f523a91c3604a1b8d1c2aced77f77ca: No bootstrap required, opened a new log
I20260810 23:30:29.969213 13491 ts_tablet_manager.cc:1397] T 56bf396f8336451e918cbac5f04d6446 P 1f523a91c3604a1b8d1c2aced77f77ca: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260810 23:30:29.970837 13495 tablet_bootstrap.cc:492] T 56bf396f8336451e918cbac5f04d6446 P ea53dd0dce004ac6b5e80f10f9f463ac: Bootstrap starting.
I20260810 23:30:29.973965 13495 tablet_bootstrap.cc:654] T 56bf396f8336451e918cbac5f04d6446 P ea53dd0dce004ac6b5e80f10f9f463ac: Neither blocks nor log segments found. Creating new log.
I20260810 23:30:29.975119 13495 log.cc:824] T 56bf396f8336451e918cbac5f04d6446 P ea53dd0dce004ac6b5e80f10f9f463ac: Log is configured to *not* fsync() on all Append() calls
I20260810 23:30:29.977248 13489 raft_consensus.cc:348] T 56bf396f8336451e918cbac5f04d6446 P c8f21a75c82642e19547c25ac88bab2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f523a91c3604a1b8d1c2aced77f77ca" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44457 } } peers { permanent_uuid: "c8f21a75c82642e19547c25ac88bab2b" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 42477 } } peers { permanent_uuid: "ea53dd0dce004ac6b5e80f10f9f463ac" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 45691 } }
I20260810 23:30:29.977737 13489 raft_consensus.cc:374] T 56bf396f8336451e918cbac5f04d6446 P c8f21a75c82642e19547c25ac88bab2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:30:29.977896 13489 raft_consensus.cc:729] T 56bf396f8336451e918cbac5f04d6446 P c8f21a75c82642e19547c25ac88bab2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f21a75c82642e19547c25ac88bab2b, State: Initialized, Role: FOLLOWER
I20260810 23:30:29.978590 13489 consensus_queue.cc:260] T 56bf396f8336451e918cbac5f04d6446 P c8f21a75c82642e19547c25ac88bab2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f523a91c3604a1b8d1c2aced77f77ca" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44457 } } peers { permanent_uuid: "c8f21a75c82642e19547c25ac88bab2b" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 42477 } } peers { permanent_uuid: "ea53dd0dce004ac6b5e80f10f9f463ac" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 45691 } }
I20260810 23:30:29.979804 13261 heartbeater.cc:503] Master 127.14.175.126:43221 was elected leader, sending a full tablet report...
I20260810 23:30:29.979946 13495 tablet_bootstrap.cc:492] T 56bf396f8336451e918cbac5f04d6446 P ea53dd0dce004ac6b5e80f10f9f463ac: No bootstrap required, opened a new log
I20260810 23:30:29.980216 13495 ts_tablet_manager.cc:1397] T 56bf396f8336451e918cbac5f04d6446 P ea53dd0dce004ac6b5e80f10f9f463ac: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260810 23:30:29.980472 13489 ts_tablet_manager.cc:1428] T 56bf396f8336451e918cbac5f04d6446 P c8f21a75c82642e19547c25ac88bab2b: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260810 23:30:29.981202 13491 raft_consensus.cc:348] T 56bf396f8336451e918cbac5f04d6446 P 1f523a91c3604a1b8d1c2aced77f77ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f523a91c3604a1b8d1c2aced77f77ca" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44457 } } peers { permanent_uuid: "c8f21a75c82642e19547c25ac88bab2b" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 42477 } } peers { permanent_uuid: "ea53dd0dce004ac6b5e80f10f9f463ac" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 45691 } }
I20260810 23:30:29.981637 13491 raft_consensus.cc:374] T 56bf396f8336451e918cbac5f04d6446 P 1f523a91c3604a1b8d1c2aced77f77ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:30:29.981746 13491 raft_consensus.cc:729] T 56bf396f8336451e918cbac5f04d6446 P 1f523a91c3604a1b8d1c2aced77f77ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f523a91c3604a1b8d1c2aced77f77ca, State: Initialized, Role: FOLLOWER
I20260810 23:30:29.982339 13491 consensus_queue.cc:260] T 56bf396f8336451e918cbac5f04d6446 P 1f523a91c3604a1b8d1c2aced77f77ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f523a91c3604a1b8d1c2aced77f77ca" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44457 } } peers { permanent_uuid: "c8f21a75c82642e19547c25ac88bab2b" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 42477 } } peers { permanent_uuid: "ea53dd0dce004ac6b5e80f10f9f463ac" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 45691 } }
I20260810 23:30:29.983384 13461 heartbeater.cc:503] Master 127.14.175.126:43221 was elected leader, sending a full tablet report...
I20260810 23:30:29.983906 13491 ts_tablet_manager.cc:1428] T 56bf396f8336451e918cbac5f04d6446 P 1f523a91c3604a1b8d1c2aced77f77ca: Time spent starting tablet: real 0.015s user 0.009s sys 0.003s
I20260810 23:30:29.991560 13495 raft_consensus.cc:348] T 56bf396f8336451e918cbac5f04d6446 P ea53dd0dce004ac6b5e80f10f9f463ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f523a91c3604a1b8d1c2aced77f77ca" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44457 } } peers { permanent_uuid: "c8f21a75c82642e19547c25ac88bab2b" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 42477 } } peers { permanent_uuid: "ea53dd0dce004ac6b5e80f10f9f463ac" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 45691 } }
I20260810 23:30:29.991969 13495 raft_consensus.cc:374] T 56bf396f8336451e918cbac5f04d6446 P ea53dd0dce004ac6b5e80f10f9f463ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:30:29.992074 13495 raft_consensus.cc:729] T 56bf396f8336451e918cbac5f04d6446 P ea53dd0dce004ac6b5e80f10f9f463ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea53dd0dce004ac6b5e80f10f9f463ac, State: Initialized, Role: FOLLOWER
I20260810 23:30:29.992707 13495 consensus_queue.cc:260] T 56bf396f8336451e918cbac5f04d6446 P ea53dd0dce004ac6b5e80f10f9f463ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f523a91c3604a1b8d1c2aced77f77ca" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44457 } } peers { permanent_uuid: "c8f21a75c82642e19547c25ac88bab2b" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 42477 } } peers { permanent_uuid: "ea53dd0dce004ac6b5e80f10f9f463ac" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 45691 } }
I20260810 23:30:29.994277 13495 ts_tablet_manager.cc:1428] T 56bf396f8336451e918cbac5f04d6446 P ea53dd0dce004ac6b5e80f10f9f463ac: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
W20260810 23:30:30.112051 13462 tablet.cc:2369] T 56bf396f8336451e918cbac5f04d6446 P 1f523a91c3604a1b8d1c2aced77f77ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:30:30.131745 13262 tablet.cc:2369] T 56bf396f8336451e918cbac5f04d6446 P c8f21a75c82642e19547c25ac88bab2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:30:30.163360 13055 tablet.cc:2369] T 56bf396f8336451e918cbac5f04d6446 P ea53dd0dce004ac6b5e80f10f9f463ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:30:30.397104 13498 raft_consensus.cc:482] T 56bf396f8336451e918cbac5f04d6446 P c8f21a75c82642e19547c25ac88bab2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:30:30.397472 13498 raft_consensus.cc:504] T 56bf396f8336451e918cbac5f04d6446 P c8f21a75c82642e19547c25ac88bab2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f523a91c3604a1b8d1c2aced77f77ca" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44457 } } peers { permanent_uuid: "c8f21a75c82642e19547c25ac88bab2b" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 42477 } } peers { permanent_uuid: "ea53dd0dce004ac6b5e80f10f9f463ac" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 45691 } }
I20260810 23:30:30.399271 13498 leader_election.cc:288] T 56bf396f8336451e918cbac5f04d6446 P c8f21a75c82642e19547c25ac88bab2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f523a91c3604a1b8d1c2aced77f77ca (127.14.175.67:44457), ea53dd0dce004ac6b5e80f10f9f463ac (127.14.175.65:45691)
I20260810 23:30:30.405777 13502 raft_consensus.cc:482] T 56bf396f8336451e918cbac5f04d6446 P ea53dd0dce004ac6b5e80f10f9f463ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:30:30.406229 13488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:30.400108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:41589 (local address 127.14.175.65:45691)
0810 23:30:30.400369 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:30.400375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:30.400398 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:30:30.400573 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:30.400578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:30.400673 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:30.400764 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:30:30.401515 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:30.402421 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:30.404406 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:30.405299 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:30.405461 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:30.405531 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:30.405834 (+ 303us) server_negotiation.cc:299] Negotiation successful
0810 23:30:30.405967 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260810 23:30:30.406294 13504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:30.399592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:45691 (local address 127.14.175.66:41589)
0810 23:30:30.400279 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:30.400296 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:30.400414 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:30.400929 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:30.400935 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:30.400964 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:30:30.401347 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:30.401361 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:30.402561 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:30.402567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:30.404200 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:30.404213 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:30.405483 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:30.405491 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:30.405673 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:30.405702 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 23:30:30.405785 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 23:30:30.405927 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":1,"thread_start_us":384,"threads_started":1}
I20260810 23:30:30.406154 13502 raft_consensus.cc:504] T 56bf396f8336451e918cbac5f04d6446 P ea53dd0dce004ac6b5e80f10f9f463ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f523a91c3604a1b8d1c2aced77f77ca" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44457 } } peers { permanent_uuid: "c8f21a75c82642e19547c25ac88bab2b" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 42477 } } peers { permanent_uuid: "ea53dd0dce004ac6b5e80f10f9f463ac" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 45691 } }
I20260810 23:30:30.406463 13483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:30.399713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:41437 (local address 127.14.175.67:44457)
0810 23:30:30.399914 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:30.399920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:30.400982 (+ 1062us) server_negotiation.cc:407] Connection header received
0810 23:30:30.401205 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:30.401210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:30.401288 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:30.401383 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:30:30.402016 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:30.402824 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:30.404585 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:30.405401 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:30.405552 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:30.405622 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:30.406079 (+ 457us) server_negotiation.cc:299] Negotiation successful
0810 23:30:30.406230 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 23:30:30.407013 13505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:30.399680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:44457 (local address 127.14.175.66:41437)
0810 23:30:30.400882 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:30.400906 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:30.401033 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:30.401402 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:30.401406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:30.401430 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:30:30.401857 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:30.401870 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:30.402952 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:30.402955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:30.404378 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:30.404392 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:30.405687 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:30.405694 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:30.405866 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:30.405900 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 23:30:30.406603 (+ 703us) client_negotiation.cc:240] Negotiation successful
0810 23:30:30.406743 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1063,"thread_start_us":860,"threads_started":1}
I20260810 23:30:30.407603 12985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56bf396f8336451e918cbac5f04d6446" candidate_uuid: "c8f21a75c82642e19547c25ac88bab2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea53dd0dce004ac6b5e80f10f9f463ac" is_pre_election: true
I20260810 23:30:30.407923 13389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56bf396f8336451e918cbac5f04d6446" candidate_uuid: "c8f21a75c82642e19547c25ac88bab2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f523a91c3604a1b8d1c2aced77f77ca" is_pre_election: true
I20260810 23:30:30.408106 12985 raft_consensus.cc:2454] T 56bf396f8336451e918cbac5f04d6446 P ea53dd0dce004ac6b5e80f10f9f463ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f21a75c82642e19547c25ac88bab2b in term 0.
I20260810 23:30:30.408463 13389 raft_consensus.cc:2454] T 56bf396f8336451e918cbac5f04d6446 P 1f523a91c3604a1b8d1c2aced77f77ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f21a75c82642e19547c25ac88bab2b in term 0.
I20260810 23:30:30.408555 13502 leader_election.cc:288] T 56bf396f8336451e918cbac5f04d6446 P ea53dd0dce004ac6b5e80f10f9f463ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f523a91c3604a1b8d1c2aced77f77ca (127.14.175.67:44457), c8f21a75c82642e19547c25ac88bab2b (127.14.175.66:42477)
I20260810 23:30:30.408987 13087 leader_election.cc:302] T 56bf396f8336451e918cbac5f04d6446 P c8f21a75c82642e19547c25ac88bab2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8f21a75c82642e19547c25ac88bab2b, ea53dd0dce004ac6b5e80f10f9f463ac; no voters:
I20260810 23:30:30.409723 13498 raft_consensus.cc:2781] T 56bf396f8336451e918cbac5f04d6446 P c8f21a75c82642e19547c25ac88bab2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:30:30.409888 13498 raft_consensus.cc:482] T 56bf396f8336451e918cbac5f04d6446 P c8f21a75c82642e19547c25ac88bab2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:30:30.409976 13498 raft_consensus.cc:3037] T 56bf396f8336451e918cbac5f04d6446 P c8f21a75c82642e19547c25ac88bab2b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:30:30.413347 13498 raft_consensus.cc:504] T 56bf396f8336451e918cbac5f04d6446 P c8f21a75c82642e19547c25ac88bab2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f523a91c3604a1b8d1c2aced77f77ca" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44457 } } peers { permanent_uuid: "c8f21a75c82642e19547c25ac88bab2b" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 42477 } } peers { permanent_uuid: "ea53dd0dce004ac6b5e80f10f9f463ac" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 45691 } }
I20260810 23:30:30.414566 12985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56bf396f8336451e918cbac5f04d6446" candidate_uuid: "c8f21a75c82642e19547c25ac88bab2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea53dd0dce004ac6b5e80f10f9f463ac"
I20260810 23:30:30.414552 13389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56bf396f8336451e918cbac5f04d6446" candidate_uuid: "c8f21a75c82642e19547c25ac88bab2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f523a91c3604a1b8d1c2aced77f77ca"
I20260810 23:30:30.414907 13389 raft_consensus.cc:3037] T 56bf396f8336451e918cbac5f04d6446 P 1f523a91c3604a1b8d1c2aced77f77ca [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:30:30.414911 12985 raft_consensus.cc:3037] T 56bf396f8336451e918cbac5f04d6446 P ea53dd0dce004ac6b5e80f10f9f463ac [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:30:30.417647 13498 leader_election.cc:288] T 56bf396f8336451e918cbac5f04d6446 P c8f21a75c82642e19547c25ac88bab2b [CANDIDATE]: Term 1 election: Requested vote from peers 1f523a91c3604a1b8d1c2aced77f77ca (127.14.175.67:44457), ea53dd0dce004ac6b5e80f10f9f463ac (127.14.175.65:45691)
I20260810 23:30:30.417989 13483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:30.409417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:60273 (local address 127.14.175.67:44457)
0810 23:30:30.410740 (+ 1323us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:30.410746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:30.410763 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:30:30.410836 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:30.410841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:30.410908 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:30.410988 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:30:30.411761 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:30.412524 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:30.415314 (+ 2790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:30.416654 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:30.416807 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:30.416883 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:30.417505 (+ 622us) server_negotiation.cc:299] Negotiation successful
0810 23:30:30.417672 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1187}
I20260810 23:30:30.418286 13486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:30.409574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:36103 (local address 127.14.175.66:42477)
0810 23:30:30.409763 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:30.409770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:30.410406 (+ 636us) server_negotiation.cc:407] Connection header received
0810 23:30:30.410640 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:30.410645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:30.410745 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:30.410839 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:30:30.411731 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:30.412789 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:30.415367 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:30.416598 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:30.417269 (+ 671us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:30.417342 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:30.417870 (+ 528us) server_negotiation.cc:299] Negotiation successful
0810 23:30:30.418016 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 23:30:30.418708 13506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:30.408778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:44457 (local address 127.14.175.65:60273)
0810 23:30:30.409990 (+ 1212us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:30.410014 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:30.410188 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:30.411120 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:30.411129 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:30.411149 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:30:30.411592 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:30.411601 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:30.412677 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:30.412683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:30.415041 (+ 2358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:30.415064 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:30.417093 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:30.417100 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:30.417307 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:30.417341 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 23:30:30.418190 (+ 849us) client_negotiation.cc:240] Negotiation successful
0810 23:30:30.418349 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1072,"mutex_wait_us":34,"thread_start_us":872,"threads_started":1}
I20260810 23:30:30.418908 13507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:30.409311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:42477 (local address 127.14.175.65:36103)
0810 23:30:30.410322 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:30.410339 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:30.410499 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:30.411036 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:30.411040 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:30.411067 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:30:30.411524 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:30.411540 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:30.412925 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:30.412931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:30.415148 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:30.415160 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:30.417514 (+ 2354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:30.417520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:30.417682 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:30.417712 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 23:30:30.418450 (+ 738us) client_negotiation.cc:240] Negotiation successful
0810 23:30:30.418581 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"mutex_wait_us":1,"thread_start_us":711,"threads_started":1}
I20260810 23:30:30.419239 12985 raft_consensus.cc:2454] T 56bf396f8336451e918cbac5f04d6446 P ea53dd0dce004ac6b5e80f10f9f463ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f21a75c82642e19547c25ac88bab2b in term 1.
I20260810 23:30:30.419566 13387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56bf396f8336451e918cbac5f04d6446" candidate_uuid: "ea53dd0dce004ac6b5e80f10f9f463ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f523a91c3604a1b8d1c2aced77f77ca" is_pre_election: true
I20260810 23:30:30.419925 13087 leader_election.cc:302] T 56bf396f8336451e918cbac5f04d6446 P c8f21a75c82642e19547c25ac88bab2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8f21a75c82642e19547c25ac88bab2b, ea53dd0dce004ac6b5e80f10f9f463ac; no voters:
I20260810 23:30:30.419946 13389 raft_consensus.cc:2454] T 56bf396f8336451e918cbac5f04d6446 P 1f523a91c3604a1b8d1c2aced77f77ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f21a75c82642e19547c25ac88bab2b in term 1.
I20260810 23:30:30.420282 13498 raft_consensus.cc:2781] T 56bf396f8336451e918cbac5f04d6446 P c8f21a75c82642e19547c25ac88bab2b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:30:30.420637 13195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56bf396f8336451e918cbac5f04d6446" candidate_uuid: "ea53dd0dce004ac6b5e80f10f9f463ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8f21a75c82642e19547c25ac88bab2b" is_pre_election: true
I20260810 23:30:30.420933 13498 raft_consensus.cc:686] T 56bf396f8336451e918cbac5f04d6446 P c8f21a75c82642e19547c25ac88bab2b [term 1 LEADER]: Becoming Leader. State: Replica: c8f21a75c82642e19547c25ac88bab2b, State: Running, Role: LEADER
I20260810 23:30:30.421471 13498 consensus_queue.cc:237] T 56bf396f8336451e918cbac5f04d6446 P c8f21a75c82642e19547c25ac88bab2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f523a91c3604a1b8d1c2aced77f77ca" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44457 } } peers { permanent_uuid: "c8f21a75c82642e19547c25ac88bab2b" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 42477 } } peers { permanent_uuid: "ea53dd0dce004ac6b5e80f10f9f463ac" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 45691 } }
I20260810 23:30:30.425043 12880 leader_election.cc:302] T 56bf396f8336451e918cbac5f04d6446 P ea53dd0dce004ac6b5e80f10f9f463ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea53dd0dce004ac6b5e80f10f9f463ac; no voters: 1f523a91c3604a1b8d1c2aced77f77ca, c8f21a75c82642e19547c25ac88bab2b
I20260810 23:30:30.425558 13502 raft_consensus.cc:2726] T 56bf396f8336451e918cbac5f04d6446 P ea53dd0dce004ac6b5e80f10f9f463ac [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:30:30.426174 12755 catalog_manager.cc:5495] T 56bf396f8336451e918cbac5f04d6446 P c8f21a75c82642e19547c25ac88bab2b reported cstate change: term changed from 0 to 1, leader changed from <none> to c8f21a75c82642e19547c25ac88bab2b (127.14.175.66). New cstate: current_term: 1 leader_uuid: "c8f21a75c82642e19547c25ac88bab2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f523a91c3604a1b8d1c2aced77f77ca" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f21a75c82642e19547c25ac88bab2b" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 42477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea53dd0dce004ac6b5e80f10f9f463ac" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 45691 } health_report { overall_health: UNKNOWN } } }
I20260810 23:30:30.506018 13486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:30.499819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45948 (local address 127.14.175.66:42477)
0810 23:30:30.499985 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:30.499990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:30.500531 (+ 541us) server_negotiation.cc:407] Connection header received
0810 23:30:30.500797 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:30.500802 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:30.500907 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:30.501071 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:30:30.501591 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:30.502345 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:30.503795 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:30.504205 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:30.504299 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:30.504807 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:30:30.505380 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:30:30.505516 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:30.505715 (+ 199us) server_negotiation.cc:299] Negotiation successful
0810 23:30:30.505760 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:30:30.514652 13387 raft_consensus.cc:1261] T 56bf396f8336451e918cbac5f04d6446 P 1f523a91c3604a1b8d1c2aced77f77ca [term 1 FOLLOWER]: Refusing update from remote peer c8f21a75c82642e19547c25ac88bab2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:30:30.514717 12985 raft_consensus.cc:1261] T 56bf396f8336451e918cbac5f04d6446 P ea53dd0dce004ac6b5e80f10f9f463ac [term 1 FOLLOWER]: Refusing update from remote peer c8f21a75c82642e19547c25ac88bab2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:30:30.515632 13508 consensus_queue.cc:1035] T 56bf396f8336451e918cbac5f04d6446 P c8f21a75c82642e19547c25ac88bab2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f523a91c3604a1b8d1c2aced77f77ca" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:30:30.516047 13498 consensus_queue.cc:1035] T 56bf396f8336451e918cbac5f04d6446 P c8f21a75c82642e19547c25ac88bab2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea53dd0dce004ac6b5e80f10f9f463ac" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:30:30.542871 12754 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:30:30.543162 12754 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:30:30.547348 12754 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 80403775157b4f28817d61569ef3cfd0: Sending DeleteTablet for 3 replicas of tablet 56bf396f8336451e918cbac5f04d6446
I20260810 23:30:30.548872 13163 tablet_service.cc:1514] Processing DeleteTablet for tablet 56bf396f8336451e918cbac5f04d6446 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:30:30 UTC) from {username='slave'} at 127.0.0.1:45934
I20260810 23:30:30.549331 12956 tablet_service.cc:1514] Processing DeleteTablet for tablet 56bf396f8336451e918cbac5f04d6446 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:30:30 UTC) from {username='slave'} at 127.0.0.1:58764
I20260810 23:30:30.549741 13363 tablet_service.cc:1514] Processing DeleteTablet for tablet 56bf396f8336451e918cbac5f04d6446 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:30:30 UTC) from {username='slave'} at 127.0.0.1:43356
I20260810 23:30:30.550869 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 12856
I20260810 23:30:30.560309 13537 tablet_replica.cc:330] T 56bf396f8336451e918cbac5f04d6446 P 1f523a91c3604a1b8d1c2aced77f77ca: stopping tablet replica
I20260810 23:30:30.561764 13537 raft_consensus.cc:2229] T 56bf396f8336451e918cbac5f04d6446 P 1f523a91c3604a1b8d1c2aced77f77ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:30:30.563287 13537 raft_consensus.cc:2258] T 56bf396f8336451e918cbac5f04d6446 P 1f523a91c3604a1b8d1c2aced77f77ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:30:30.564108 13533 tablet_replica.cc:330] T 56bf396f8336451e918cbac5f04d6446 P c8f21a75c82642e19547c25ac88bab2b: stopping tablet replica
I20260810 23:30:30.565022 13534 tablet_replica.cc:330] T 56bf396f8336451e918cbac5f04d6446 P ea53dd0dce004ac6b5e80f10f9f463ac: stopping tablet replica
I20260810 23:30:30.565585 13534 raft_consensus.cc:2229] T 56bf396f8336451e918cbac5f04d6446 P ea53dd0dce004ac6b5e80f10f9f463ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:30:30.565572 13533 raft_consensus.cc:2229] T 56bf396f8336451e918cbac5f04d6446 P c8f21a75c82642e19547c25ac88bab2b [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:30:30.566310 13534 raft_consensus.cc:2258] T 56bf396f8336451e918cbac5f04d6446 P ea53dd0dce004ac6b5e80f10f9f463ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:30:30.567811 13537 ts_tablet_manager.cc:1905] T 56bf396f8336451e918cbac5f04d6446 P 1f523a91c3604a1b8d1c2aced77f77ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:30:30.568420 13534 ts_tablet_manager.cc:1905] T 56bf396f8336451e918cbac5f04d6446 P ea53dd0dce004ac6b5e80f10f9f463ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:30:30.568641 13533 raft_consensus.cc:2258] T 56bf396f8336451e918cbac5f04d6446 P c8f21a75c82642e19547c25ac88bab2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:30:30.570715 13533 ts_tablet_manager.cc:1905] T 56bf396f8336451e918cbac5f04d6446 P c8f21a75c82642e19547c25ac88bab2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:30:30.576445 13534 ts_tablet_manager.cc:1918] T 56bf396f8336451e918cbac5f04d6446 P ea53dd0dce004ac6b5e80f10f9f463ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:30:30.576820 13534 log.cc:1196] T 56bf396f8336451e918cbac5f04d6446 P ea53dd0dce004ac6b5e80f10f9f463ac: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-0/wal/wals/56bf396f8336451e918cbac5f04d6446
I20260810 23:30:30.577493 13534 ts_tablet_manager.cc:1939] T 56bf396f8336451e918cbac5f04d6446 P ea53dd0dce004ac6b5e80f10f9f463ac: Deleting consensus metadata
I20260810 23:30:30.578703 13533 ts_tablet_manager.cc:1918] T 56bf396f8336451e918cbac5f04d6446 P c8f21a75c82642e19547c25ac88bab2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:30:30.578992 13533 log.cc:1196] T 56bf396f8336451e918cbac5f04d6446 P c8f21a75c82642e19547c25ac88bab2b: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-1/wal/wals/56bf396f8336451e918cbac5f04d6446
I20260810 23:30:30.579653 13533 ts_tablet_manager.cc:1939] T 56bf396f8336451e918cbac5f04d6446 P c8f21a75c82642e19547c25ac88bab2b: Deleting consensus metadata
I20260810 23:30:30.579787 12736 catalog_manager.cc:4841] TS ea53dd0dce004ac6b5e80f10f9f463ac (127.14.175.65:45691): tablet 56bf396f8336451e918cbac5f04d6446 (table test-table [id=d590b3103c4e47a38cc650a56e50db86]) successfully deleted
I20260810 23:30:30.580614 13537 ts_tablet_manager.cc:1918] T 56bf396f8336451e918cbac5f04d6446 P 1f523a91c3604a1b8d1c2aced77f77ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:30:30.580833 13537 log.cc:1196] T 56bf396f8336451e918cbac5f04d6446 P 1f523a91c3604a1b8d1c2aced77f77ca: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404299514076-15037-0/minicluster-data/ts-2/wal/wals/56bf396f8336451e918cbac5f04d6446
I20260810 23:30:30.581441 13537 ts_tablet_manager.cc:1939] T 56bf396f8336451e918cbac5f04d6446 P 1f523a91c3604a1b8d1c2aced77f77ca: Deleting consensus metadata
I20260810 23:30:30.581739 12735 catalog_manager.cc:4841] TS c8f21a75c82642e19547c25ac88bab2b (127.14.175.66:42477): tablet 56bf396f8336451e918cbac5f04d6446 (table test-table [id=d590b3103c4e47a38cc650a56e50db86]) successfully deleted
I20260810 23:30:30.583077 13488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:30.552553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58768 (local address 127.14.175.65:45691)
0810 23:30:30.552763 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:30.552769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:30.564044 (+ 11275us) server_negotiation.cc:407] Connection header received
0810 23:30:30.564315 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:30.564322 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:30.564480 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:30.564616 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:30.565474 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:30.566820 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:30.568800 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:30.569521 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:30.569636 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:30.580977 (+ 11341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:30.580991 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:30.581011 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:30.581911 (+ 900us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:30.581984 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:30.582018 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:30.582130 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:30.582610 (+ 480us) server_negotiation.cc:299] Negotiation successful
0810 23:30:30.582740 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 23:30:30.583501 12734 catalog_manager.cc:4841] TS 1f523a91c3604a1b8d1c2aced77f77ca (127.14.175.67:44457): tablet 56bf396f8336451e918cbac5f04d6446 (table test-table [id=d590b3103c4e47a38cc650a56e50db86]) successfully deleted
I20260810 23:30:30.583865 13048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:30:30.713193 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 12856
I20260810 23:30:30.746809 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 13061
I20260810 23:30:30.754796 13486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:30.747732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45960 (local address 127.14.175.66:42477)
0810 23:30:30.747914 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:30.747920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:30.747944 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:30:30.748045 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:30.748049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:30.748138 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:30.748238 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:30.748731 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:30.749609 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:30.751455 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:30.751919 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:30.752014 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:30.753279 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:30.753290 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:30.753307 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:30.753967 (+ 660us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:30.754027 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:30.754052 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:30.754166 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:30.754319 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 23:30:30.754418 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:30:30.755550 13252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:30:30.877492 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 13061
I20260810 23:30:30.901081 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 13265
I20260810 23:30:30.907805 13483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:30.901869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43372 (local address 127.14.175.67:44457)
0810 23:30:30.902034 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:30.902038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:30.902110 (+ 72us) server_negotiation.cc:407] Connection header received
0810 23:30:30.902257 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:30.902261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:30.902363 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:30.902450 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:30.902837 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:30.903543 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:30.904855 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:30.905397 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:30.905499 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:30.906483 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:30.906493 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:30.906511 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:30.907158 (+ 647us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:30.907210 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:30.907231 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:30.907325 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:30.907426 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 23:30:30.907517 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 23:30:30.908500 13452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:30:30.997128 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 13265
I20260810 23:30:31.022104 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 12709
I20260810 23:30:31.023211 12801 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:30:31.133273 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 12709
2026-08-10T23:30:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3386 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:30:31 dist-test-slave-rmhg krb5kdc[13550](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:30:31 dist-test-slave-rmhg krb5kdc[13550](info): set up 2 sockets
Aug 10 23:30:31 dist-test-slave-rmhg krb5kdc[13550](info): commencing operation
krb5kdc: starting...
W20260810 23:30:32.404043 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.220s user 0.006s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:30:32 dist-test-slave-rmhg krb5kdc[13550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:30:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:30:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
I20260810 23:30:32.732044 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:46239
--webserver_interface=127.14.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:45085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:46239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:30:33.087303 13570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:30:33.087630 13570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:30:33.087751 13570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:30:33.095230 13570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:30:33.095363 13570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:30:33.095417 13570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:30:33.095500 13570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:30:33.095577 13570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:30:33.106006 13570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:45085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:46239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:46239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:30:33.107466 13570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:30:33.108843 13570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:30:33.121286 13584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:33.121340 13581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:33.121286 13586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:30:33.121879 13570 server_base.cc:1034] running on GCE node
I20260810 23:30:33.122936 13570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:30:33.124492 13570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:30:33.125715 13570 hybrid_clock.cc:648] HybridClock initialized: now 1786404633125665 us; error 36 us; skew 500 ppm
I20260810 23:30:33.126080 13570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:30:33 dist-test-slave-rmhg krb5kdc[13550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404633, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:30:33.129456 13570 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:30:33.131188 13570 webserver.cc:460] Webserver started at http://127.14.175.126:33067/ using document root <none> and password file <none>
I20260810 23:30:33.131953 13570 fs_manager.cc:359] Metadata directory not provided
I20260810 23:30:33.132114 13570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:30:33.132505 13570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:30:33.136085 13570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "0e48c93436764c93b3e24df120b182f4"
format_stamp: "Formatted at 2026-08-10 23:30:33 on dist-test-slave-rmhg"
server_key: "41212fff5df07c376f9f891477f083c5"
server_key_iv: "88b45bca5e414ae57e6e2b7be54986ab"
server_key_version: "encryptionkey@0"
I20260810 23:30:33.136930 13570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "0e48c93436764c93b3e24df120b182f4"
format_stamp: "Formatted at 2026-08-10 23:30:33 on dist-test-slave-rmhg"
server_key: "41212fff5df07c376f9f891477f083c5"
server_key_iv: "88b45bca5e414ae57e6e2b7be54986ab"
server_key_version: "encryptionkey@0"
I20260810 23:30:33.142951 13570 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260810 23:30:33.147095 13598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:33.148779 13570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:30:33.149013 13570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "0e48c93436764c93b3e24df120b182f4"
format_stamp: "Formatted at 2026-08-10 23:30:33 on dist-test-slave-rmhg"
server_key: "41212fff5df07c376f9f891477f083c5"
server_key_iv: "88b45bca5e414ae57e6e2b7be54986ab"
server_key_version: "encryptionkey@0"
I20260810 23:30:33.149293 13570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:30:33.178920 13570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:30:33.183740 13570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:30:33.184060 13570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:30:33.203293 13570 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:46239
I20260810 23:30:33.203315 13680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:46239 every 8 connection(s)
I20260810 23:30:33.204816 13570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:30:33.208968 13681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:30:33.214385 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 13570
I20260810 23:30:33.214579 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:30:33.215049 15037 external_mini_cluster.cc:1428] Setting key 6b0b05d577da561d45b5a33e5ddaa9ef
I20260810 23:30:33.221071 13681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e48c93436764c93b3e24df120b182f4: Bootstrap starting.
I20260810 23:30:33.225188 13681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e48c93436764c93b3e24df120b182f4: Neither blocks nor log segments found. Creating new log.
I20260810 23:30:33.226563 13681 log.cc:824] T 00000000000000000000000000000000 P 0e48c93436764c93b3e24df120b182f4: Log is configured to *not* fsync() on all Append() calls
I20260810 23:30:33.230607 13681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e48c93436764c93b3e24df120b182f4: No bootstrap required, opened a new log
Aug 10 23:30:33 dist-test-slave-rmhg krb5kdc[13550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:30:33.238883 13681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e48c93436764c93b3e24df120b182f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e48c93436764c93b3e24df120b182f4" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46239 } }
I20260810 23:30:33.239382 13681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e48c93436764c93b3e24df120b182f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:30:33.239495 13681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e48c93436764c93b3e24df120b182f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e48c93436764c93b3e24df120b182f4, State: Initialized, Role: FOLLOWER
I20260810 23:30:33.240171 13681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e48c93436764c93b3e24df120b182f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e48c93436764c93b3e24df120b182f4" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46239 } }
I20260810 23:30:33.240401 13681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e48c93436764c93b3e24df120b182f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:30:33.240442 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:33.217136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51740 (local address 127.14.175.126:46239)
0810 23:30:33.218091 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:33.218104 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:33.218164 (+ 60us) server_negotiation.cc:407] Connection header received
0810 23:30:33.219998 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:33.220026 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:33.220497 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:33.220915 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:33.222930 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:33.224366 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:33.226242 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:33.227017 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:33.227279 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:33.232313 (+ 5034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:33.232322 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:33.232341 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:30:33.232394 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:33.235777 (+ 3383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:33.236638 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:33.236646 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:33.236654 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:33.236853 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:33.237311 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:33.237318 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:33.237322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:33.237945 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:33.238280 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:33.238733 (+ 453us) server_negotiation.cc:299] Negotiation successful
0810 23:30:33.239240 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":408,"threads_started":1}
I20260810 23:30:33.240557 13681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e48c93436764c93b3e24df120b182f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:30:33.240759 13681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e48c93436764c93b3e24df120b182f4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:30:33.243227 13681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e48c93436764c93b3e24df120b182f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e48c93436764c93b3e24df120b182f4" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46239 } }
I20260810 23:30:33.243810 13681 leader_election.cc:302] T 00000000000000000000000000000000 P 0e48c93436764c93b3e24df120b182f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e48c93436764c93b3e24df120b182f4; no voters:
I20260810 23:30:33.244392 13681 leader_election.cc:288] T 00000000000000000000000000000000 P 0e48c93436764c93b3e24df120b182f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:30:33.244536 13688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e48c93436764c93b3e24df120b182f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:30:33.245251 13688 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e48c93436764c93b3e24df120b182f4 [term 1 LEADER]: Becoming Leader. State: Replica: 0e48c93436764c93b3e24df120b182f4, State: Running, Role: LEADER
I20260810 23:30:33.245867 13688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e48c93436764c93b3e24df120b182f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e48c93436764c93b3e24df120b182f4" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46239 } }
I20260810 23:30:33.246619 13681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e48c93436764c93b3e24df120b182f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:30:33.248546 13690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e48c93436764c93b3e24df120b182f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e48c93436764c93b3e24df120b182f4. Latest consensus state: current_term: 1 leader_uuid: "0e48c93436764c93b3e24df120b182f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e48c93436764c93b3e24df120b182f4" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46239 } } }
I20260810 23:30:33.249192 13689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e48c93436764c93b3e24df120b182f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e48c93436764c93b3e24df120b182f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e48c93436764c93b3e24df120b182f4" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 46239 } } }
I20260810 23:30:33.249599 13690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e48c93436764c93b3e24df120b182f4 [sys.catalog]: This master's current role is: LEADER
I20260810 23:30:33.249600 13689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e48c93436764c93b3e24df120b182f4 [sys.catalog]: This master's current role is: LEADER
I20260810 23:30:33.252288 13693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:30:33.257762 13693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:30:33.266943 13693 catalog_manager.cc:1347] Generated new cluster ID: d0f039d3d3f348a1b3a552e175ccf2a8
I20260810 23:30:33.267077 13693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:30:33.286761 13693 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:30:33.288884 13693 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:30:33.300417 13693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e48c93436764c93b3e24df120b182f4: Generated new TSK 0
I20260810 23:30:33.301679 13693 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.175.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
Entry for principal kudu/127.14.175.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab.
I20260810 23:30:33.382033 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.65:0
--local_ip_for_outbound_sockets=127.14.175.65
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:46239
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:45085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:30:33.745694 13719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:30:33.746006 13719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:30:33.746143 13719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:30:33.754067 13719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:30:33.754200 13719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:30:33.754361 13719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.65
I20260810 23:30:33.764696 13719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:45085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.65.keytab
--principal=kudu/127.14.175.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:46239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.65
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:30:33.766196 13719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:30:33.767639 13719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:30:33.781790 13728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:33.781811 13726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:33.781998 13725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:30:33.782707 13719 server_base.cc:1034] running on GCE node
I20260810 23:30:33.783383 13719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:30:33.784497 13719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:30:33.785712 13719 hybrid_clock.cc:648] HybridClock initialized: now 1786404633785655 us; error 46 us; skew 500 ppm
I20260810 23:30:33.786060 13719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:30:33 dist-test-slave-rmhg krb5kdc[13550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404633, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:30:33.789395 13719 init.cc:377] Logged in from keytab as kudu/127.14.175.65@KRBTEST.COM (short username kudu)
I20260810 23:30:33.790980 13719 webserver.cc:460] Webserver started at http://127.14.175.65:37165/ using document root <none> and password file <none>
I20260810 23:30:33.791741 13719 fs_manager.cc:359] Metadata directory not provided
I20260810 23:30:33.791878 13719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:30:33.792215 13719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:30:33.795558 13719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-0/data/instance:
uuid: "cb0a44e20ae44d60be377e134f6dd73d"
format_stamp: "Formatted at 2026-08-10 23:30:33 on dist-test-slave-rmhg"
server_key: "e0ff39b5dc9854ee716c87a9632e4f13"
server_key_iv: "eca69577d11061b31d47906eee5a9768"
server_key_version: "encryptionkey@0"
I20260810 23:30:33.796414 13719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance:
uuid: "cb0a44e20ae44d60be377e134f6dd73d"
format_stamp: "Formatted at 2026-08-10 23:30:33 on dist-test-slave-rmhg"
server_key: "e0ff39b5dc9854ee716c87a9632e4f13"
server_key_iv: "eca69577d11061b31d47906eee5a9768"
server_key_version: "encryptionkey@0"
I20260810 23:30:33.802237 13719 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 23:30:33.806221 13741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:33.807644 13719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:30:33.807844 13719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-0/wal
uuid: "cb0a44e20ae44d60be377e134f6dd73d"
format_stamp: "Formatted at 2026-08-10 23:30:33 on dist-test-slave-rmhg"
server_key: "e0ff39b5dc9854ee716c87a9632e4f13"
server_key_iv: "eca69577d11061b31d47906eee5a9768"
server_key_version: "encryptionkey@0"
I20260810 23:30:33.808079 13719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:30:33.827199 13719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:30:33.831549 13719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:30:33.831856 13719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:30:33.832954 13719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:30:33.834502 13719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:30:33.834594 13719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:33.834731 13719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:30:33.834786 13719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:33.865913 13719 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.65:36795
I20260810 23:30:33.865952 13907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.65:36795 every 8 connection(s)
I20260810 23:30:33.867233 13719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
I20260810 23:30:33.875771 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 13719
I20260810 23:30:33.875945 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance
I20260810 23:30:33.876410 15037 external_mini_cluster.cc:1428] Setting key cad5139ff6b27ec45b46ad8349046539
Aug 10 23:30:33 dist-test-slave-rmhg krb5kdc[13550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404633, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.65@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:30:33.888841 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:33.869893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:38731 (local address 127.14.175.126:46239)
0810 23:30:33.870366 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:33.870370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:33.871402 (+ 1032us) server_negotiation.cc:407] Connection header received
0810 23:30:33.873288 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:33.873293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:33.873366 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:33.873464 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:33.875008 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:33.875740 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:33.877583 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:33.878025 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:33.878140 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:33.881835 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:33.881842 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:33.881847 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:30:33.881892 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:33.885141 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:33.886190 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:33.886197 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:33.886201 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:33.886366 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:33.886866 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:33.886871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:33.886874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:33.887236 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:33.887382 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:33.888266 (+ 884us) server_negotiation.cc:299] Negotiation successful
0810 23:30:33.888503 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":236,"threads_started":1}
I20260810 23:30:33.889948 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:33.870485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46239 (local address 127.14.175.65:38731)
0810 23:30:33.871226 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:33.871271 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:33.872999 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:33.873697 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:33.873705 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:33.874262 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:30:33.874866 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:33.874884 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:33.875904 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:33.875911 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:33.877439 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:33.877452 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:33.878288 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:33.878296 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:33.878556 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:33.879213 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:30:33.879243 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:30:33.881546 (+ 2303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:30:33.885361 (+ 3815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:30:33.885374 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:30:33.885394 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:30:33.886020 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:30:33.886470 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:30:33.886474 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:30:33.886477 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:30:33.886739 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:30:33.887352 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:30:33.887360 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:30:33.887813 (+ 453us) client_negotiation.cc:769] Sending connection context
0810 23:30:33.888111 (+ 298us) client_negotiation.cc:240] Negotiation successful
0810 23:30:33.888634 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":276,"threads_started":1}
I20260810 23:30:33.891683 13909 heartbeater.cc:347] Connected to a master server at 127.14.175.126:46239
I20260810 23:30:33.892053 13909 heartbeater.cc:464] Registering TS with master...
I20260810 23:30:33.892978 13909 heartbeater.cc:511] Master 127.14.175.126:46239 requested a full tablet report, sending...
I20260810 23:30:33.895342 13625 ts_manager.cc:194] Registered new tserver with Master: cb0a44e20ae44d60be377e134f6dd73d (127.14.175.65:36795)
I20260810 23:30:33.896895 13625 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.65@KRBTEST.COM'} at 127.14.175.65:38731
WARNING: no policy specified for kudu/127.14.175.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
Entry for principal kudu/127.14.175.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab.
I20260810 23:30:33.947940 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.66:0
--local_ip_for_outbound_sockets=127.14.175.66
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:46239
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:45085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:30:34.304275 13919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:30:34.304628 13919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:30:34.304766 13919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:30:34.312482 13919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:30:34.312644 13919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:30:34.312800 13919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.66
I20260810 23:30:34.323153 13919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:45085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.66.keytab
--principal=kudu/127.14.175.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:46239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.66
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:30:34.324564 13919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:30:34.325989 13919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:30:34.339949 13935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:34.340008 13933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:34.340184 13931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:30:34.340956 13919 server_base.cc:1034] running on GCE node
I20260810 23:30:34.341684 13919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:30:34.342794 13919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:30:34.344007 13919 hybrid_clock.cc:648] HybridClock initialized: now 1786404634343945 us; error 64 us; skew 500 ppm
I20260810 23:30:34.344333 13919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:30:34 dist-test-slave-rmhg krb5kdc[13550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404634, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:30:34.347858 13919 init.cc:377] Logged in from keytab as kudu/127.14.175.66@KRBTEST.COM (short username kudu)
I20260810 23:30:34.349692 13919 webserver.cc:460] Webserver started at http://127.14.175.66:46673/ using document root <none> and password file <none>
I20260810 23:30:34.350445 13919 fs_manager.cc:359] Metadata directory not provided
I20260810 23:30:34.350575 13919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:30:34.350915 13919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:30:34.354310 13919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-1/data/instance:
uuid: "d0e96f5610d84078b3b68311fe765f74"
format_stamp: "Formatted at 2026-08-10 23:30:34 on dist-test-slave-rmhg"
server_key: "20eb205bde84424b8f5a15dee7397095"
server_key_iv: "7cf26eaaca918b2913f565314ce26656"
server_key_version: "encryptionkey@0"
I20260810 23:30:34.355177 13919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance:
uuid: "d0e96f5610d84078b3b68311fe765f74"
format_stamp: "Formatted at 2026-08-10 23:30:34 on dist-test-slave-rmhg"
server_key: "20eb205bde84424b8f5a15dee7397095"
server_key_iv: "7cf26eaaca918b2913f565314ce26656"
server_key_version: "encryptionkey@0"
I20260810 23:30:34.360957 13919 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:30:34.364898 13945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:34.366479 13919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:30:34.366703 13919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-1/wal
uuid: "d0e96f5610d84078b3b68311fe765f74"
format_stamp: "Formatted at 2026-08-10 23:30:34 on dist-test-slave-rmhg"
server_key: "20eb205bde84424b8f5a15dee7397095"
server_key_iv: "7cf26eaaca918b2913f565314ce26656"
server_key_version: "encryptionkey@0"
I20260810 23:30:34.366940 13919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:30:34.395293 13919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:30:34.400000 13919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:30:34.400311 13919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:30:34.401423 13919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:30:34.403064 13919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:30:34.403167 13919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:34.403290 13919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:30:34.403347 13919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:34.434626 13919 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.66:33431
I20260810 23:30:34.434684 14137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.66:33431 every 8 connection(s)
I20260810 23:30:34.435992 13919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
I20260810 23:30:34.441527 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 13919
I20260810 23:30:34.441754 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance
I20260810 23:30:34.442418 15037 external_mini_cluster.cc:1428] Setting key 0ac10a71f4ae6861a5703ff4cd135abf
Aug 10 23:30:34 dist-test-slave-rmhg krb5kdc[13550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404634, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.66@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:30:34.459583 14141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:34.438861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:47201 (local address 127.14.175.126:46239)
0810 23:30:34.439372 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:34.439378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:34.440523 (+ 1145us) server_negotiation.cc:407] Connection header received
0810 23:30:34.443159 (+ 2636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:34.443166 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:34.443269 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:34.443399 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:34.445472 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:34.446271 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:34.448550 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:34.449011 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:34.449117 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:34.453428 (+ 4311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:34.453435 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:34.453438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:30:34.453487 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:34.456233 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:34.457042 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:34.457047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:34.457050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:34.457229 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:34.457717 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:34.457721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:34.457723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:34.458008 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:34.458148 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:34.458925 (+ 777us) server_negotiation.cc:299] Negotiation successful
0810 23:30:34.459174 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":244,"threads_started":1}
I20260810 23:30:34.460630 14143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:34.439483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46239 (local address 127.14.175.66:47201)
0810 23:30:34.440327 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:34.440371 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:34.442740 (+ 2369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:34.443738 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:34.443750 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:34.444378 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:30:34.445274 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:34.445305 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:34.446428 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:34.446433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:34.448371 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:34.448394 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:34.449283 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:34.449291 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:34.449586 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:34.450374 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:30:34.450406 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:30:34.453115 (+ 2709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:30:34.456397 (+ 3282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:30:34.456407 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:30:34.456431 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:30:34.456901 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:30:34.457352 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:30:34.457357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:30:34.457361 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:30:34.457582 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:30:34.458138 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:30:34.458148 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:30:34.458530 (+ 382us) client_negotiation.cc:769] Sending connection context
0810 23:30:34.458787 (+ 257us) client_negotiation.cc:240] Negotiation successful
0810 23:30:34.459393 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":325,"threads_started":1}
I20260810 23:30:34.462234 14138 heartbeater.cc:347] Connected to a master server at 127.14.175.126:46239
I20260810 23:30:34.462602 14138 heartbeater.cc:464] Registering TS with master...
I20260810 23:30:34.463508 14138 heartbeater.cc:511] Master 127.14.175.126:46239 requested a full tablet report, sending...
I20260810 23:30:34.465392 13625 ts_manager.cc:194] Registered new tserver with Master: d0e96f5610d84078b3b68311fe765f74 (127.14.175.66:33431)
I20260810 23:30:34.466516 13625 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.66@KRBTEST.COM'} at 127.14.175.66:47201
WARNING: no policy specified for kudu/127.14.175.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
Entry for principal kudu/127.14.175.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab.
I20260810 23:30:34.516732 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.67:0
--local_ip_for_outbound_sockets=127.14.175.67
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:46239
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:45085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:30:34.889880 14149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:30:34.890195 14149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:30:34.890295 14149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:30:34.897759 14149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:30:34.897923 14149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:30:34.898089 14149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.67
I20260810 23:30:34.900259 13909 heartbeater.cc:503] Master 127.14.175.126:46239 was elected leader, sending a full tablet report...
I20260810 23:30:34.908843 14149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:45085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.67.keytab
--principal=kudu/127.14.175.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:46239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.67
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:30:34.910259 14149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:30:34.911679 14149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:30:34.925572 14155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:34.925602 14160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:34.925544 14158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:30:34.926378 14149 server_base.cc:1034] running on GCE node
I20260810 23:30:34.927011 14149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:30:34.928011 14149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:30:34.929237 14149 hybrid_clock.cc:648] HybridClock initialized: now 1786404634929185 us; error 35 us; skew 500 ppm
I20260810 23:30:34.929563 14149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:30:34 dist-test-slave-rmhg krb5kdc[13550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404634, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:30:34.932812 14149 init.cc:377] Logged in from keytab as kudu/127.14.175.67@KRBTEST.COM (short username kudu)
I20260810 23:30:34.934414 14149 webserver.cc:460] Webserver started at http://127.14.175.67:44185/ using document root <none> and password file <none>
I20260810 23:30:34.935182 14149 fs_manager.cc:359] Metadata directory not provided
I20260810 23:30:34.935319 14149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:30:34.935659 14149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:30:34.938983 14149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-2/data/instance:
uuid: "d26f18c9fb484ef8b55609748532ecf9"
format_stamp: "Formatted at 2026-08-10 23:30:34 on dist-test-slave-rmhg"
server_key: "2de85beea704e7b88463f175bcd0c168"
server_key_iv: "9d6eebbcfa1e1eff6cc7c442bb8f65f9"
server_key_version: "encryptionkey@0"
I20260810 23:30:34.939872 14149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance:
uuid: "d26f18c9fb484ef8b55609748532ecf9"
format_stamp: "Formatted at 2026-08-10 23:30:34 on dist-test-slave-rmhg"
server_key: "2de85beea704e7b88463f175bcd0c168"
server_key_iv: "9d6eebbcfa1e1eff6cc7c442bb8f65f9"
server_key_version: "encryptionkey@0"
I20260810 23:30:34.945924 14149 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260810 23:30:34.950116 14169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:34.951685 14149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260810 23:30:34.951887 14149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-2/wal
uuid: "d26f18c9fb484ef8b55609748532ecf9"
format_stamp: "Formatted at 2026-08-10 23:30:34 on dist-test-slave-rmhg"
server_key: "2de85beea704e7b88463f175bcd0c168"
server_key_iv: "9d6eebbcfa1e1eff6cc7c442bb8f65f9"
server_key_version: "encryptionkey@0"
I20260810 23:30:34.952091 14149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:30:34.967674 14149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:30:34.971951 14149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:30:34.972245 14149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:30:34.973291 14149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:30:34.974717 14149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:30:34.974797 14149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:34.974915 14149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:30:34.974974 14149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:35.005462 14149 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.67:37043
I20260810 23:30:35.005518 14344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.67:37043 every 8 connection(s)
I20260810 23:30:35.006841 14149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
I20260810 23:30:35.010306 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 14149
I20260810 23:30:35.010490 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance
I20260810 23:30:35.010972 15037 external_mini_cluster.cc:1428] Setting key 07c271c48d2ecd92ae49db5f96faeb42
Aug 10 23:30:35 dist-test-slave-rmhg krb5kdc[13550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404634, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.67@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:30:35.027271 14349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:35.009538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:53533 (local address 127.14.175.126:46239)
0810 23:30:35.010078 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:35.010084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:35.011144 (+ 1060us) server_negotiation.cc:407] Connection header received
0810 23:30:35.013365 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:35.013370 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:35.013455 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:35.013542 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:35.015179 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:35.015954 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:35.017502 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:35.017896 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:35.017989 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:35.021293 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:35.021298 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:35.021301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:30:35.021334 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:35.023890 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:35.024671 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:35.024677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:35.024680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:35.024838 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:35.025390 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:35.025394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:35.025397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:35.025743 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:35.025904 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:35.026794 (+ 890us) server_negotiation.cc:299] Negotiation successful
0810 23:30:35.026968 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":237,"threads_started":1}
I20260810 23:30:35.028241 14351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:35.010136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:46239 (local address 127.14.175.67:53533)
0810 23:30:35.010952 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:35.011002 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:35.013041 (+ 2039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:35.013844 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:35.013856 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:35.014424 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:30:35.015018 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:35.015037 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:35.016077 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:35.016080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:35.017388 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:35.017400 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:35.018070 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:35.018074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:35.018268 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:35.018907 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:30:35.018930 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:30:35.021070 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:30:35.024029 (+ 2959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:30:35.024039 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:30:35.024054 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:30:35.024531 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:30:35.024961 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:30:35.024966 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:30:35.024969 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:30:35.025264 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:30:35.025905 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:30:35.025913 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:30:35.026384 (+ 471us) client_negotiation.cc:769] Sending connection context
0810 23:30:35.026694 (+ 310us) client_negotiation.cc:240] Negotiation successful
0810 23:30:35.027126 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":323,"threads_started":1}
I20260810 23:30:35.029578 14347 heartbeater.cc:347] Connected to a master server at 127.14.175.126:46239
I20260810 23:30:35.029878 14347 heartbeater.cc:464] Registering TS with master...
I20260810 23:30:35.030596 14347 heartbeater.cc:511] Master 127.14.175.126:46239 requested a full tablet report, sending...
I20260810 23:30:35.031970 13625 ts_manager.cc:194] Registered new tserver with Master: d26f18c9fb484ef8b55609748532ecf9 (127.14.175.67:37043)
I20260810 23:30:35.032773 13625 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.175.67@KRBTEST.COM'} at 127.14.175.67:53533
I20260810 23:30:35.038937 15037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:30:35.056438 14349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:35.043210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51756 (local address 127.14.175.126:46239)
0810 23:30:35.043395 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:35.043399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:35.043747 (+ 348us) server_negotiation.cc:407] Connection header received
0810 23:30:35.043868 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:35.043872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:35.043943 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:35.044032 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:35.045710 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:35.046476 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:35.047883 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:35.048412 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:35.048511 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:35.050947 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:35.050953 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:35.050956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:30:35.050994 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:35.053824 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:35.054520 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:35.054524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:35.054527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:35.054693 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:35.055040 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:35.055043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:35.055046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:35.055394 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:35.055591 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:35.055967 (+ 376us) server_negotiation.cc:299] Negotiation successful
0810 23:30:35.056185 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 23:30:35.060439 13625 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:30:35.067531 13625 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:30:35.103950 14364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:35.080515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:37043 (local address 127.0.0.1:47236)
0810 23:30:35.093949 (+ 13434us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:35.094019 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:35.094286 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:35.095519 (+ 1233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:35.095528 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:35.095576 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:30:35.096067 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:35.096082 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:35.098112 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:35.098117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:35.100837 (+ 2720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:35.100863 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:35.102755 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:35.102761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:35.102955 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:35.103015 (+ 60us) client_negotiation.cc:769] Sending connection context
0810 23:30:35.103292 (+ 277us) client_negotiation.cc:240] Negotiation successful
0810 23:30:35.103517 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13053,"mutex_wait_us":32,"spinlock_wait_cycles":4480,"thread_start_us":14134,"threads_started":1}
I20260810 23:30:35.104102 14366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:35.081012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48496 (local address 127.14.175.65:36795)
0810 23:30:35.093682 (+ 12670us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:35.093695 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:35.094622 (+ 927us) server_negotiation.cc:407] Connection header received
0810 23:30:35.094845 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:35.094853 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:35.095054 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:35.095242 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:30:35.096904 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:35.098467 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:35.101033 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:35.102545 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:35.102697 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:35.102924 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:35.103561 (+ 637us) server_negotiation.cc:299] Negotiation successful
0810 23:30:35.103770 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12107,"thread_start_us":12149,"threads_started":1}
I20260810 23:30:35.104751 14363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:35.080716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:36795 (local address 127.0.0.1:48496)
0810 23:30:35.094459 (+ 13743us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:35.094482 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:35.094632 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:35.095519 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:35.095528 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:35.095571 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:30:35.096689 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:35.096702 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:35.098579 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:35.098585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:35.100837 (+ 2252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:35.100863 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:35.103083 (+ 2220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:35.103090 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:35.103337 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:35.103372 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 23:30:35.104183 (+ 811us) client_negotiation.cc:240] Negotiation successful
0810 23:30:35.104315 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13619,"mutex_wait_us":639,"spinlock_wait_cycles":1408,"thread_start_us":13475,"threads_started":1}
I20260810 23:30:35.104974 14367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:35.095321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:33431 (local address 127.0.0.1:32912)
0810 23:30:35.096594 (+ 1273us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:35.096621 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:35.096731 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:35.097871 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:35.097875 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:35.097905 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:30:35.098326 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:35.098339 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:35.100113 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:35.100117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:35.101981 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:35.102022 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:35.103856 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:35.103863 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:35.104049 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:35.104359 (+ 310us) client_negotiation.cc:769] Sending connection context
0810 23:30:35.104454 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 23:30:35.104610 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1128,"thread_start_us":946,"threads_started":1}
I20260810 23:30:35.105355 14365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:35.080744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47236 (local address 127.14.175.67:37043)
0810 23:30:35.094276 (+ 13532us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:35.094284 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:35.094328 (+ 44us) server_negotiation.cc:407] Connection header received
0810 23:30:35.094633 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:35.094643 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:35.094946 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:35.095169 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:30:35.096582 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:35.097927 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:35.101072 (+ 3145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:35.102113 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:35.102294 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:35.102569 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:35.104498 (+ 1929us) server_negotiation.cc:299] Negotiation successful
0810 23:30:35.104687 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13352,"thread_start_us":13172,"threads_started":1}
I20260810 23:30:35.105602 14368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:35.095700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32912 (local address 127.14.175.66:33431)
0810 23:30:35.097245 (+ 1545us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:35.097255 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:35.097284 (+ 29us) server_negotiation.cc:407] Connection header received
0810 23:30:35.097383 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:35.097391 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:35.097647 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:35.097848 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:30:35.098726 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:35.099968 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:35.102217 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:35.103634 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:35.103775 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:35.103956 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:35.104547 (+ 591us) server_negotiation.cc:299] Negotiation successful
0810 23:30:35.104712 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1336,"thread_start_us":1210,"threads_started":1}
I20260810 23:30:35.108989 13812 tablet_service.cc:1467] Processing CreateTablet for tablet 3d1f8168a4d24aac881d591be9b66620 (DEFAULT_TABLE table=test-table [id=b59a214011b94e04994c612e95807e1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:30:35.109016 14241 tablet_service.cc:1467] Processing CreateTablet for tablet 3d1f8168a4d24aac881d591be9b66620 (DEFAULT_TABLE table=test-table [id=b59a214011b94e04994c612e95807e1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:30:35.109061 14022 tablet_service.cc:1467] Processing CreateTablet for tablet 3d1f8168a4d24aac881d591be9b66620 (DEFAULT_TABLE table=test-table [id=b59a214011b94e04994c612e95807e1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:30:35.110466 13812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d1f8168a4d24aac881d591be9b66620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:30:35.110509 14241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d1f8168a4d24aac881d591be9b66620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:30:35.110510 14022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d1f8168a4d24aac881d591be9b66620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:30:35.123744 14369 tablet_bootstrap.cc:492] T 3d1f8168a4d24aac881d591be9b66620 P cb0a44e20ae44d60be377e134f6dd73d: Bootstrap starting.
I20260810 23:30:35.127934 14369 tablet_bootstrap.cc:654] T 3d1f8168a4d24aac881d591be9b66620 P cb0a44e20ae44d60be377e134f6dd73d: Neither blocks nor log segments found. Creating new log.
I20260810 23:30:35.129097 14369 log.cc:824] T 3d1f8168a4d24aac881d591be9b66620 P cb0a44e20ae44d60be377e134f6dd73d: Log is configured to *not* fsync() on all Append() calls
I20260810 23:30:35.130219 14370 tablet_bootstrap.cc:492] T 3d1f8168a4d24aac881d591be9b66620 P d26f18c9fb484ef8b55609748532ecf9: Bootstrap starting.
I20260810 23:30:35.131265 14371 tablet_bootstrap.cc:492] T 3d1f8168a4d24aac881d591be9b66620 P d0e96f5610d84078b3b68311fe765f74: Bootstrap starting.
I20260810 23:30:35.133984 14369 tablet_bootstrap.cc:492] T 3d1f8168a4d24aac881d591be9b66620 P cb0a44e20ae44d60be377e134f6dd73d: No bootstrap required, opened a new log
I20260810 23:30:35.134279 14369 ts_tablet_manager.cc:1397] T 3d1f8168a4d24aac881d591be9b66620 P cb0a44e20ae44d60be377e134f6dd73d: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260810 23:30:35.135275 14370 tablet_bootstrap.cc:654] T 3d1f8168a4d24aac881d591be9b66620 P d26f18c9fb484ef8b55609748532ecf9: Neither blocks nor log segments found. Creating new log.
I20260810 23:30:35.135267 14371 tablet_bootstrap.cc:654] T 3d1f8168a4d24aac881d591be9b66620 P d0e96f5610d84078b3b68311fe765f74: Neither blocks nor log segments found. Creating new log.
I20260810 23:30:35.136636 14370 log.cc:824] T 3d1f8168a4d24aac881d591be9b66620 P d26f18c9fb484ef8b55609748532ecf9: Log is configured to *not* fsync() on all Append() calls
I20260810 23:30:35.137035 14371 log.cc:824] T 3d1f8168a4d24aac881d591be9b66620 P d0e96f5610d84078b3b68311fe765f74: Log is configured to *not* fsync() on all Append() calls
I20260810 23:30:35.139721 14370 tablet_bootstrap.cc:492] T 3d1f8168a4d24aac881d591be9b66620 P d26f18c9fb484ef8b55609748532ecf9: No bootstrap required, opened a new log
I20260810 23:30:35.140002 14370 ts_tablet_manager.cc:1397] T 3d1f8168a4d24aac881d591be9b66620 P d26f18c9fb484ef8b55609748532ecf9: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260810 23:30:35.140244 14371 tablet_bootstrap.cc:492] T 3d1f8168a4d24aac881d591be9b66620 P d0e96f5610d84078b3b68311fe765f74: No bootstrap required, opened a new log
I20260810 23:30:35.140574 14371 ts_tablet_manager.cc:1397] T 3d1f8168a4d24aac881d591be9b66620 P d0e96f5610d84078b3b68311fe765f74: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260810 23:30:35.144723 14369 raft_consensus.cc:348] T 3d1f8168a4d24aac881d591be9b66620 P cb0a44e20ae44d60be377e134f6dd73d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26f18c9fb484ef8b55609748532ecf9" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37043 } } peers { permanent_uuid: "cb0a44e20ae44d60be377e134f6dd73d" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 36795 } } peers { permanent_uuid: "d0e96f5610d84078b3b68311fe765f74" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 33431 } }
I20260810 23:30:35.145304 14369 raft_consensus.cc:374] T 3d1f8168a4d24aac881d591be9b66620 P cb0a44e20ae44d60be377e134f6dd73d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:30:35.145409 14369 raft_consensus.cc:729] T 3d1f8168a4d24aac881d591be9b66620 P cb0a44e20ae44d60be377e134f6dd73d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb0a44e20ae44d60be377e134f6dd73d, State: Initialized, Role: FOLLOWER
I20260810 23:30:35.146137 14369 consensus_queue.cc:260] T 3d1f8168a4d24aac881d591be9b66620 P cb0a44e20ae44d60be377e134f6dd73d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26f18c9fb484ef8b55609748532ecf9" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37043 } } peers { permanent_uuid: "cb0a44e20ae44d60be377e134f6dd73d" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 36795 } } peers { permanent_uuid: "d0e96f5610d84078b3b68311fe765f74" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 33431 } }
I20260810 23:30:35.148110 14370 raft_consensus.cc:348] T 3d1f8168a4d24aac881d591be9b66620 P d26f18c9fb484ef8b55609748532ecf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26f18c9fb484ef8b55609748532ecf9" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37043 } } peers { permanent_uuid: "cb0a44e20ae44d60be377e134f6dd73d" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 36795 } } peers { permanent_uuid: "d0e96f5610d84078b3b68311fe765f74" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 33431 } }
I20260810 23:30:35.148710 14370 raft_consensus.cc:374] T 3d1f8168a4d24aac881d591be9b66620 P d26f18c9fb484ef8b55609748532ecf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:30:35.148833 14370 raft_consensus.cc:729] T 3d1f8168a4d24aac881d591be9b66620 P d26f18c9fb484ef8b55609748532ecf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d26f18c9fb484ef8b55609748532ecf9, State: Initialized, Role: FOLLOWER
I20260810 23:30:35.149914 14369 ts_tablet_manager.cc:1428] T 3d1f8168a4d24aac881d591be9b66620 P cb0a44e20ae44d60be377e134f6dd73d: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260810 23:30:35.150637 14370 consensus_queue.cc:260] T 3d1f8168a4d24aac881d591be9b66620 P d26f18c9fb484ef8b55609748532ecf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26f18c9fb484ef8b55609748532ecf9" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37043 } } peers { permanent_uuid: "cb0a44e20ae44d60be377e134f6dd73d" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 36795 } } peers { permanent_uuid: "d0e96f5610d84078b3b68311fe765f74" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 33431 } }
I20260810 23:30:35.151314 14371 raft_consensus.cc:348] T 3d1f8168a4d24aac881d591be9b66620 P d0e96f5610d84078b3b68311fe765f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26f18c9fb484ef8b55609748532ecf9" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37043 } } peers { permanent_uuid: "cb0a44e20ae44d60be377e134f6dd73d" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 36795 } } peers { permanent_uuid: "d0e96f5610d84078b3b68311fe765f74" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 33431 } }
I20260810 23:30:35.151787 14347 heartbeater.cc:503] Master 127.14.175.126:46239 was elected leader, sending a full tablet report...
I20260810 23:30:35.151815 14371 raft_consensus.cc:374] T 3d1f8168a4d24aac881d591be9b66620 P d0e96f5610d84078b3b68311fe765f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:30:35.151914 14371 raft_consensus.cc:729] T 3d1f8168a4d24aac881d591be9b66620 P d0e96f5610d84078b3b68311fe765f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e96f5610d84078b3b68311fe765f74, State: Initialized, Role: FOLLOWER
I20260810 23:30:35.152572 14370 ts_tablet_manager.cc:1428] T 3d1f8168a4d24aac881d591be9b66620 P d26f18c9fb484ef8b55609748532ecf9: Time spent starting tablet: real 0.012s user 0.007s sys 0.007s
I20260810 23:30:35.152683 14371 consensus_queue.cc:260] T 3d1f8168a4d24aac881d591be9b66620 P d0e96f5610d84078b3b68311fe765f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26f18c9fb484ef8b55609748532ecf9" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37043 } } peers { permanent_uuid: "cb0a44e20ae44d60be377e134f6dd73d" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 36795 } } peers { permanent_uuid: "d0e96f5610d84078b3b68311fe765f74" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 33431 } }
I20260810 23:30:35.154165 14138 heartbeater.cc:503] Master 127.14.175.126:46239 was elected leader, sending a full tablet report...
I20260810 23:30:35.154886 14371 ts_tablet_manager.cc:1428] T 3d1f8168a4d24aac881d591be9b66620 P d0e96f5610d84078b3b68311fe765f74: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
W20260810 23:30:35.188079 14140 tablet.cc:2369] T 3d1f8168a4d24aac881d591be9b66620 P d0e96f5610d84078b3b68311fe765f74: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:30:35.258467 14348 tablet.cc:2369] T 3d1f8168a4d24aac881d591be9b66620 P d26f18c9fb484ef8b55609748532ecf9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:30:35.273779 14379 raft_consensus.cc:482] T 3d1f8168a4d24aac881d591be9b66620 P cb0a44e20ae44d60be377e134f6dd73d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:30:35.274120 14379 raft_consensus.cc:504] T 3d1f8168a4d24aac881d591be9b66620 P cb0a44e20ae44d60be377e134f6dd73d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26f18c9fb484ef8b55609748532ecf9" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37043 } } peers { permanent_uuid: "cb0a44e20ae44d60be377e134f6dd73d" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 36795 } } peers { permanent_uuid: "d0e96f5610d84078b3b68311fe765f74" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 33431 } }
I20260810 23:30:35.275945 14379 leader_election.cc:288] T 3d1f8168a4d24aac881d591be9b66620 P cb0a44e20ae44d60be377e134f6dd73d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d26f18c9fb484ef8b55609748532ecf9 (127.14.175.67:37043), d0e96f5610d84078b3b68311fe765f74 (127.14.175.66:33431)
I20260810 23:30:35.283192 14365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:35.276426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:45055 (local address 127.14.175.67:37043)
0810 23:30:35.276623 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:35.276628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:35.277609 (+ 981us) server_negotiation.cc:407] Connection header received
0810 23:30:35.277771 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:35.277777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:35.277855 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:35.277949 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:30:35.278623 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:35.279419 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:35.281334 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:35.282132 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:35.282288 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:35.282368 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:35.282802 (+ 434us) server_negotiation.cc:299] Negotiation successful
0810 23:30:35.282933 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 23:30:35.283212 14368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:35.276486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:57695 (local address 127.14.175.66:33431)
0810 23:30:35.276714 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:35.276720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:35.277787 (+ 1067us) server_negotiation.cc:407] Connection header received
0810 23:30:35.278007 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:35.278013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:35.278090 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:35.278204 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:30:35.278789 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:35.279491 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:35.281352 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:35.282158 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:35.282309 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:35.282403 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:35.282815 (+ 412us) server_negotiation.cc:299] Negotiation successful
0810 23:30:35.282946 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 23:30:35.283341 14382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:35.276510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:37043 (local address 127.14.175.65:45055)
0810 23:30:35.277470 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:35.277488 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:35.277631 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:35.278015 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:35.278019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:35.278047 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:30:35.278466 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:35.278478 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:35.279555 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:35.279560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:35.281177 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:35.281191 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:35.282390 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:35.282398 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:35.282611 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:35.282667 (+ 56us) client_negotiation.cc:769] Sending connection context
0810 23:30:35.282750 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 23:30:35.282889 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"mutex_wait_us":38,"thread_start_us":459,"threads_started":1}
I20260810 23:30:35.283387 14384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:35.276511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:33431 (local address 127.14.175.65:57695)
0810 23:30:35.277670 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:35.277688 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:35.277831 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:35.278230 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:35.278233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:35.278253 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:30:35.278609 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:35.278621 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:35.279626 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:35.279630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:35.281174 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:35.281192 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:35.282440 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:35.282447 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:35.282626 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:35.282677 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 23:30:35.282761 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 23:30:35.282892 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1029,"mutex_wait_us":184,"thread_start_us":576,"threads_started":1}
I20260810 23:30:35.284564 14274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1f8168a4d24aac881d591be9b66620" candidate_uuid: "cb0a44e20ae44d60be377e134f6dd73d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d26f18c9fb484ef8b55609748532ecf9" is_pre_election: true
I20260810 23:30:35.284564 14053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1f8168a4d24aac881d591be9b66620" candidate_uuid: "cb0a44e20ae44d60be377e134f6dd73d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e96f5610d84078b3b68311fe765f74" is_pre_election: true
I20260810 23:30:35.285229 14274 raft_consensus.cc:2454] T 3d1f8168a4d24aac881d591be9b66620 P d26f18c9fb484ef8b55609748532ecf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb0a44e20ae44d60be377e134f6dd73d in term 0.
I20260810 23:30:35.285229 14053 raft_consensus.cc:2454] T 3d1f8168a4d24aac881d591be9b66620 P d0e96f5610d84078b3b68311fe765f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb0a44e20ae44d60be377e134f6dd73d in term 0.
I20260810 23:30:35.286024 13746 leader_election.cc:302] T 3d1f8168a4d24aac881d591be9b66620 P cb0a44e20ae44d60be377e134f6dd73d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb0a44e20ae44d60be377e134f6dd73d, d0e96f5610d84078b3b68311fe765f74; no voters:
I20260810 23:30:35.286477 14379 raft_consensus.cc:2781] T 3d1f8168a4d24aac881d591be9b66620 P cb0a44e20ae44d60be377e134f6dd73d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:30:35.286623 14379 raft_consensus.cc:482] T 3d1f8168a4d24aac881d591be9b66620 P cb0a44e20ae44d60be377e134f6dd73d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:30:35.286757 14379 raft_consensus.cc:3037] T 3d1f8168a4d24aac881d591be9b66620 P cb0a44e20ae44d60be377e134f6dd73d [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:30:35.289242 14379 raft_consensus.cc:504] T 3d1f8168a4d24aac881d591be9b66620 P cb0a44e20ae44d60be377e134f6dd73d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26f18c9fb484ef8b55609748532ecf9" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37043 } } peers { permanent_uuid: "cb0a44e20ae44d60be377e134f6dd73d" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 36795 } } peers { permanent_uuid: "d0e96f5610d84078b3b68311fe765f74" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 33431 } }
I20260810 23:30:35.289927 14379 leader_election.cc:288] T 3d1f8168a4d24aac881d591be9b66620 P cb0a44e20ae44d60be377e134f6dd73d [CANDIDATE]: Term 1 election: Requested vote from peers d26f18c9fb484ef8b55609748532ecf9 (127.14.175.67:37043), d0e96f5610d84078b3b68311fe765f74 (127.14.175.66:33431)
I20260810 23:30:35.290405 14274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1f8168a4d24aac881d591be9b66620" candidate_uuid: "cb0a44e20ae44d60be377e134f6dd73d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d26f18c9fb484ef8b55609748532ecf9"
I20260810 23:30:35.290417 14053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1f8168a4d24aac881d591be9b66620" candidate_uuid: "cb0a44e20ae44d60be377e134f6dd73d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e96f5610d84078b3b68311fe765f74"
I20260810 23:30:35.290602 14053 raft_consensus.cc:3037] T 3d1f8168a4d24aac881d591be9b66620 P d0e96f5610d84078b3b68311fe765f74 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:30:35.290596 14274 raft_consensus.cc:3037] T 3d1f8168a4d24aac881d591be9b66620 P d26f18c9fb484ef8b55609748532ecf9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:30:35.292804 14274 raft_consensus.cc:2454] T 3d1f8168a4d24aac881d591be9b66620 P d26f18c9fb484ef8b55609748532ecf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb0a44e20ae44d60be377e134f6dd73d in term 1.
I20260810 23:30:35.292809 14053 raft_consensus.cc:2454] T 3d1f8168a4d24aac881d591be9b66620 P d0e96f5610d84078b3b68311fe765f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb0a44e20ae44d60be377e134f6dd73d in term 1.
I20260810 23:30:35.293254 13746 leader_election.cc:302] T 3d1f8168a4d24aac881d591be9b66620 P cb0a44e20ae44d60be377e134f6dd73d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: cb0a44e20ae44d60be377e134f6dd73d, d0e96f5610d84078b3b68311fe765f74, d26f18c9fb484ef8b55609748532ecf9; no voters:
I20260810 23:30:35.293566 14379 raft_consensus.cc:2781] T 3d1f8168a4d24aac881d591be9b66620 P cb0a44e20ae44d60be377e134f6dd73d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:30:35.294010 14379 raft_consensus.cc:686] T 3d1f8168a4d24aac881d591be9b66620 P cb0a44e20ae44d60be377e134f6dd73d [term 1 LEADER]: Becoming Leader. State: Replica: cb0a44e20ae44d60be377e134f6dd73d, State: Running, Role: LEADER
I20260810 23:30:35.294505 14379 consensus_queue.cc:237] T 3d1f8168a4d24aac881d591be9b66620 P cb0a44e20ae44d60be377e134f6dd73d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26f18c9fb484ef8b55609748532ecf9" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37043 } } peers { permanent_uuid: "cb0a44e20ae44d60be377e134f6dd73d" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 36795 } } peers { permanent_uuid: "d0e96f5610d84078b3b68311fe765f74" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 33431 } }
I20260810 23:30:35.299510 13624 catalog_manager.cc:5495] T 3d1f8168a4d24aac881d591be9b66620 P cb0a44e20ae44d60be377e134f6dd73d reported cstate change: term changed from 0 to 1, leader changed from <none> to cb0a44e20ae44d60be377e134f6dd73d (127.14.175.65). New cstate: current_term: 1 leader_uuid: "cb0a44e20ae44d60be377e134f6dd73d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26f18c9fb484ef8b55609748532ecf9" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb0a44e20ae44d60be377e134f6dd73d" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 36795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0e96f5610d84078b3b68311fe765f74" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 33431 } health_report { overall_health: UNKNOWN } } }
I20260810 23:30:35.348023 14366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:35.342665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48498 (local address 127.14.175.65:36795)
0810 23:30:35.342836 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:35.342842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:35.342990 (+ 148us) server_negotiation.cc:407] Connection header received
0810 23:30:35.343131 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:35.343135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:35.343249 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:35.343386 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:30:35.343732 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:35.344499 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:35.345895 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:35.346315 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:35.346407 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:35.346963 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:30:35.347529 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:30:35.347643 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:35.347713 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 23:30:35.347800 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 23:30:35.357059 14274 raft_consensus.cc:1261] T 3d1f8168a4d24aac881d591be9b66620 P d26f18c9fb484ef8b55609748532ecf9 [term 1 FOLLOWER]: Refusing update from remote peer cb0a44e20ae44d60be377e134f6dd73d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:30:35.357096 14053 raft_consensus.cc:1261] T 3d1f8168a4d24aac881d591be9b66620 P d0e96f5610d84078b3b68311fe765f74 [term 1 FOLLOWER]: Refusing update from remote peer cb0a44e20ae44d60be377e134f6dd73d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:30:35.358174 14379 consensus_queue.cc:1035] T 3d1f8168a4d24aac881d591be9b66620 P cb0a44e20ae44d60be377e134f6dd73d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d26f18c9fb484ef8b55609748532ecf9" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:30:35.358520 14385 consensus_queue.cc:1035] T 3d1f8168a4d24aac881d591be9b66620 P cb0a44e20ae44d60be377e134f6dd73d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e96f5610d84078b3b68311fe765f74" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 33431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 23:30:35.370052 13910 tablet.cc:2369] T 3d1f8168a4d24aac881d591be9b66620 P cb0a44e20ae44d60be377e134f6dd73d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:30:35.370580 14388 mvcc.cc:205] Tried to move back new op lower bound from 7317113386411126784 to 7317113386169978880. Current Snapshot: MvccSnapshot[applied={T|T < 7317113386411126784}]
I20260810 23:30:35.394743 13624 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:30:35.395076 13624 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:30:35.399758 13624 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e48c93436764c93b3e24df120b182f4: Sending DeleteTablet for 3 replicas of tablet 3d1f8168a4d24aac881d591be9b66620
I20260810 23:30:35.401051 14241 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d1f8168a4d24aac881d591be9b66620 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:30:35 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:47236
I20260810 23:30:35.402113 13812 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d1f8168a4d24aac881d591be9b66620 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:30:35 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:48496
I20260810 23:30:35.403095 14404 tablet_replica.cc:330] T 3d1f8168a4d24aac881d591be9b66620 P cb0a44e20ae44d60be377e134f6dd73d: stopping tablet replica
I20260810 23:30:35.403231 14022 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d1f8168a4d24aac881d591be9b66620 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:30:35 UTC) from {username='kudu', principal='kudu/127.14.175.126@KRBTEST.COM'} at 127.0.0.1:32912
I20260810 23:30:35.403587 14404 raft_consensus.cc:2229] T 3d1f8168a4d24aac881d591be9b66620 P cb0a44e20ae44d60be377e134f6dd73d [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:30:35.404328 14404 raft_consensus.cc:2258] T 3d1f8168a4d24aac881d591be9b66620 P cb0a44e20ae44d60be377e134f6dd73d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:30:35.405055 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 13719
I20260810 23:30:35.406924 14404 ts_tablet_manager.cc:1905] T 3d1f8168a4d24aac881d591be9b66620 P cb0a44e20ae44d60be377e134f6dd73d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:30:35.410928 14405 tablet_replica.cc:330] T 3d1f8168a4d24aac881d591be9b66620 P d0e96f5610d84078b3b68311fe765f74: stopping tablet replica
I20260810 23:30:35.412211 14405 raft_consensus.cc:2229] T 3d1f8168a4d24aac881d591be9b66620 P d0e96f5610d84078b3b68311fe765f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:30:35.413586 14405 raft_consensus.cc:2258] T 3d1f8168a4d24aac881d591be9b66620 P d0e96f5610d84078b3b68311fe765f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:30:35.417135 14405 ts_tablet_manager.cc:1905] T 3d1f8168a4d24aac881d591be9b66620 P d0e96f5610d84078b3b68311fe765f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:30:35.418413 14404 ts_tablet_manager.cc:1918] T 3d1f8168a4d24aac881d591be9b66620 P cb0a44e20ae44d60be377e134f6dd73d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:30:35.418625 14404 log.cc:1196] T 3d1f8168a4d24aac881d591be9b66620 P cb0a44e20ae44d60be377e134f6dd73d: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-0/wal/wals/3d1f8168a4d24aac881d591be9b66620
I20260810 23:30:35.418934 14403 tablet_replica.cc:330] T 3d1f8168a4d24aac881d591be9b66620 P d26f18c9fb484ef8b55609748532ecf9: stopping tablet replica
I20260810 23:30:35.419231 14404 ts_tablet_manager.cc:1939] T 3d1f8168a4d24aac881d591be9b66620 P cb0a44e20ae44d60be377e134f6dd73d: Deleting consensus metadata
I20260810 23:30:35.419606 14403 raft_consensus.cc:2229] T 3d1f8168a4d24aac881d591be9b66620 P d26f18c9fb484ef8b55609748532ecf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:30:35.421332 13606 catalog_manager.cc:4841] TS cb0a44e20ae44d60be377e134f6dd73d (127.14.175.65:36795): tablet 3d1f8168a4d24aac881d591be9b66620 (table test-table [id=b59a214011b94e04994c612e95807e1e]) successfully deleted
I20260810 23:30:35.422168 14403 raft_consensus.cc:2258] T 3d1f8168a4d24aac881d591be9b66620 P d26f18c9fb484ef8b55609748532ecf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:30:35.424773 14403 ts_tablet_manager.cc:1905] T 3d1f8168a4d24aac881d591be9b66620 P d26f18c9fb484ef8b55609748532ecf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:30:35.428357 14405 ts_tablet_manager.cc:1918] T 3d1f8168a4d24aac881d591be9b66620 P d0e96f5610d84078b3b68311fe765f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:30:35.428690 14405 log.cc:1196] T 3d1f8168a4d24aac881d591be9b66620 P d0e96f5610d84078b3b68311fe765f74: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-1/wal/wals/3d1f8168a4d24aac881d591be9b66620
I20260810 23:30:35.429344 14405 ts_tablet_manager.cc:1939] T 3d1f8168a4d24aac881d591be9b66620 P d0e96f5610d84078b3b68311fe765f74: Deleting consensus metadata
I20260810 23:30:35.432760 13606 catalog_manager.cc:4841] TS d0e96f5610d84078b3b68311fe765f74 (127.14.175.66:33431): tablet 3d1f8168a4d24aac881d591be9b66620 (table test-table [id=b59a214011b94e04994c612e95807e1e]) successfully deleted
I20260810 23:30:35.435840 14403 ts_tablet_manager.cc:1918] T 3d1f8168a4d24aac881d591be9b66620 P d26f18c9fb484ef8b55609748532ecf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:30:35.436085 14403 log.cc:1196] T 3d1f8168a4d24aac881d591be9b66620 P d26f18c9fb484ef8b55609748532ecf9: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404299514076-15037-0/minicluster-data/ts-2/wal/wals/3d1f8168a4d24aac881d591be9b66620
I20260810 23:30:35.436635 14403 ts_tablet_manager.cc:1939] T 3d1f8168a4d24aac881d591be9b66620 P d26f18c9fb484ef8b55609748532ecf9: Deleting consensus metadata
Aug 10 23:30:35 dist-test-slave-rmhg krb5kdc[13550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.65@KRBTEST.COM
I20260810 23:30:35.438684 13601 catalog_manager.cc:4841] TS d26f18c9fb484ef8b55609748532ecf9 (127.14.175.67:37043): tablet 3d1f8168a4d24aac881d591be9b66620 (table test-table [id=b59a214011b94e04994c612e95807e1e]) successfully deleted
I20260810 23:30:35.448980 14366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:35.406456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48508 (local address 127.14.175.65:36795)
0810 23:30:35.406645 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:35.406652 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:35.419272 (+ 12620us) server_negotiation.cc:407] Connection header received
0810 23:30:35.419845 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:35.419851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:35.419939 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:35.420024 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:35.422556 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:35.423807 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:35.429093 (+ 5286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:35.429943 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:35.430111 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:35.440481 (+ 10370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:35.440508 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:35.440530 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:30:35.440589 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:35.444999 (+ 4410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:35.446366 (+ 1367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:35.446377 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:35.446381 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:35.446606 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:35.447091 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:35.447096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:35.447100 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:35.447614 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:35.448012 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:35.448299 (+ 287us) server_negotiation.cc:299] Negotiation successful
0810 23:30:35.448575 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:30:35.449852 13902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:30:35.552781 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 13719
I20260810 23:30:35.581760 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 13919
Aug 10 23:30:35 dist-test-slave-rmhg krb5kdc[13550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.66@KRBTEST.COM
I20260810 23:30:35.602689 14368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:35.582537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32928 (local address 127.14.175.66:33431)
0810 23:30:35.582723 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:35.582729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:35.582749 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:30:35.582930 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:35.582934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:35.583009 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:35.583096 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:35.584483 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:35.585331 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:35.586618 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:35.587064 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:35.587208 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:35.596617 (+ 9409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:35.596638 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:35.596657 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:30:35.596700 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:35.599636 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:35.600436 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:35.600445 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:35.600447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:35.600585 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:35.600981 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:35.600986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:35.600988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:35.601470 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:35.601759 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:35.602102 (+ 343us) server_negotiation.cc:299] Negotiation successful
0810 23:30:35.602333 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 23:30:35.603407 14129 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:30:35.694871 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 13919
I20260810 23:30:35.724447 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 14149
Aug 10 23:30:35 dist-test-slave-rmhg krb5kdc[13550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.67@KRBTEST.COM
I20260810 23:30:35.741389 14365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:35.725360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47252 (local address 127.14.175.67:37043)
0810 23:30:35.725540 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:35.725545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:35.725566 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:30:35.725705 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:35.725709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:35.725789 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:35.725876 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:35.727277 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:35.728048 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:35.729503 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:35.729954 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:35.730079 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:35.734788 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:35.734810 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:35.734829 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:30:35.734874 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:35.737937 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:35.738878 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:35.738888 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:35.738890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:35.739107 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:35.739732 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:35.739736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:35.739739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:35.740174 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:35.740420 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:35.740805 (+ 385us) server_negotiation.cc:299] Negotiation successful
0810 23:30:35.741004 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 23:30:35.742086 14339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:30:35.831399 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 14149
I20260810 23:30:35.858876 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 13570
I20260810 23:30:35.860157 13675 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:30:35.974709 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 13570
2026-08-10T23:30:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4845 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T23:30:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:30:36Z Disabled control of system clock
I20260810 23:30:36.078176 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:35275
--webserver_interface=127.14.175.126
--webserver_port=0
--builtin_ntp_servers=127.14.175.84:46721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:35275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:30:36.439607 14424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:30:36.439929 14424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:30:36.440025 14424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:30:36.447628 14424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:30:36.447775 14424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:30:36.447816 14424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:30:36.447875 14424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:30:36.447921 14424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:30:36.457665 14424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:46721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:35275
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:35275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:30:36.459146 14424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:30:36.461102 14424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:30:36.473548 14439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:36.473588 14432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:36.473512 14436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:30:36.474552 14424 server_base.cc:1034] running on GCE node
I20260810 23:30:36.475363 14424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:30:36.476846 14424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:30:36.478046 14424 hybrid_clock.cc:648] HybridClock initialized: now 1786404636478001 us; error 36 us; skew 500 ppm
I20260810 23:30:36.478395 14424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:30:36.480727 14424 webserver.cc:460] Webserver started at http://127.14.175.126:45881/ using document root <none> and password file <none>
I20260810 23:30:36.481493 14424 fs_manager.cc:359] Metadata directory not provided
I20260810 23:30:36.481590 14424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:30:36.481873 14424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:30:36.484897 14424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "9f62d50aa1324f15b10d0410c05a3053"
format_stamp: "Formatted at 2026-08-10 23:30:36 on dist-test-slave-rmhg"
server_key: "a30a8db59aa4ef6a28193484fcb8f0e8"
server_key_iv: "ba6c146fd1af8c1f39fbe34967b4de3a"
server_key_version: "encryptionkey@0"
I20260810 23:30:36.485772 14424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "9f62d50aa1324f15b10d0410c05a3053"
format_stamp: "Formatted at 2026-08-10 23:30:36 on dist-test-slave-rmhg"
server_key: "a30a8db59aa4ef6a28193484fcb8f0e8"
server_key_iv: "ba6c146fd1af8c1f39fbe34967b4de3a"
server_key_version: "encryptionkey@0"
I20260810 23:30:36.491421 14424 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:30:36.495338 14451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:36.496804 14424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:30:36.496976 14424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "9f62d50aa1324f15b10d0410c05a3053"
format_stamp: "Formatted at 2026-08-10 23:30:36 on dist-test-slave-rmhg"
server_key: "a30a8db59aa4ef6a28193484fcb8f0e8"
server_key_iv: "ba6c146fd1af8c1f39fbe34967b4de3a"
server_key_version: "encryptionkey@0"
I20260810 23:30:36.497242 14424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:30:36.510408 14424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:30:36.511271 14424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:30:36.511520 14424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:30:36.530151 14424 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:35275
I20260810 23:30:36.530176 14532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:35275 every 8 connection(s)
I20260810 23:30:36.531596 14424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:30:36.535542 14533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:30:36.540731 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 14424
I20260810 23:30:36.540912 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:30:36.541431 15037 external_mini_cluster.cc:1428] Setting key 8920a79fb08ec54002331eaed692dac2
I20260810 23:30:36.547062 14533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f62d50aa1324f15b10d0410c05a3053: Bootstrap starting.
I20260810 23:30:36.550429 14533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f62d50aa1324f15b10d0410c05a3053: Neither blocks nor log segments found. Creating new log.
I20260810 23:30:36.551544 14533 log.cc:824] T 00000000000000000000000000000000 P 9f62d50aa1324f15b10d0410c05a3053: Log is configured to *not* fsync() on all Append() calls
I20260810 23:30:36.554677 14533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f62d50aa1324f15b10d0410c05a3053: No bootstrap required, opened a new log
I20260810 23:30:36.555442 14539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:36.543430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55436 (local address 127.14.175.126:35275)
0810 23:30:36.544575 (+ 1145us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:36.544591 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:36.544639 (+ 48us) server_negotiation.cc:407] Connection header received
0810 23:30:36.546482 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:36.546511 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:36.546999 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:36.547402 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:36.548005 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:36.549424 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:36.550849 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:36.551593 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:36.551769 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:36.552906 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:36.552914 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:36.552931 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:36.553562 (+ 631us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:36.553648 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:36.553678 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:36.553785 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:36.554018 (+ 233us) server_negotiation.cc:299] Negotiation successful
0810 23:30:36.554348 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":397,"threads_started":1}
I20260810 23:30:36.563158 14533 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f62d50aa1324f15b10d0410c05a3053 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f62d50aa1324f15b10d0410c05a3053" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 35275 } }
I20260810 23:30:36.563606 14533 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f62d50aa1324f15b10d0410c05a3053 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:30:36.563719 14533 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f62d50aa1324f15b10d0410c05a3053 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f62d50aa1324f15b10d0410c05a3053, State: Initialized, Role: FOLLOWER
I20260810 23:30:36.564435 14533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f62d50aa1324f15b10d0410c05a3053 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f62d50aa1324f15b10d0410c05a3053" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 35275 } }
I20260810 23:30:36.564787 14533 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f62d50aa1324f15b10d0410c05a3053 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:30:36.564953 14533 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f62d50aa1324f15b10d0410c05a3053 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:30:36.565200 14533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f62d50aa1324f15b10d0410c05a3053 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:30:36.567582 14533 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f62d50aa1324f15b10d0410c05a3053 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f62d50aa1324f15b10d0410c05a3053" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 35275 } }
I20260810 23:30:36.568104 14533 leader_election.cc:302] T 00000000000000000000000000000000 P 9f62d50aa1324f15b10d0410c05a3053 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f62d50aa1324f15b10d0410c05a3053; no voters:
I20260810 23:30:36.568720 14533 leader_election.cc:288] T 00000000000000000000000000000000 P 9f62d50aa1324f15b10d0410c05a3053 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:30:36.568909 14541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f62d50aa1324f15b10d0410c05a3053 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:30:36.569635 14541 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f62d50aa1324f15b10d0410c05a3053 [term 1 LEADER]: Becoming Leader. State: Replica: 9f62d50aa1324f15b10d0410c05a3053, State: Running, Role: LEADER
I20260810 23:30:36.570189 14541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f62d50aa1324f15b10d0410c05a3053 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f62d50aa1324f15b10d0410c05a3053" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 35275 } }
I20260810 23:30:36.570878 14533 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f62d50aa1324f15b10d0410c05a3053 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:30:36.572896 14543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f62d50aa1324f15b10d0410c05a3053 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f62d50aa1324f15b10d0410c05a3053. Latest consensus state: current_term: 1 leader_uuid: "9f62d50aa1324f15b10d0410c05a3053" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f62d50aa1324f15b10d0410c05a3053" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 35275 } } }
I20260810 23:30:36.573370 14543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f62d50aa1324f15b10d0410c05a3053 [sys.catalog]: This master's current role is: LEADER
I20260810 23:30:36.573617 14542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f62d50aa1324f15b10d0410c05a3053 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f62d50aa1324f15b10d0410c05a3053" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f62d50aa1324f15b10d0410c05a3053" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 35275 } } }
I20260810 23:30:36.574119 14542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f62d50aa1324f15b10d0410c05a3053 [sys.catalog]: This master's current role is: LEADER
I20260810 23:30:36.575603 14547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:30:36.581440 14547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:30:36.591306 14547 catalog_manager.cc:1347] Generated new cluster ID: 36b9ecf2fe7746509652dbb66106ef17
I20260810 23:30:36.591454 14547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:30:36.606437 14547 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:30:36.607837 14547 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:30:36.618695 14547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f62d50aa1324f15b10d0410c05a3053: Generated new TSK 0
I20260810 23:30:36.619714 14547 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:30:36.627764 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.65:0
--local_ip_for_outbound_sockets=127.14.175.65
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:35275
--builtin_ntp_servers=127.14.175.84:46721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:30:37.001489 14565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:30:37.001822 14565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:30:37.001917 14565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:30:37.010067 14565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:30:37.010205 14565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:30:37.010350 14565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.65
I20260810 23:30:37.020044 14565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:46721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.175.65
--webserver_port=0
--tserver_master_addrs=127.14.175.126:35275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.65
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:30:37.021602 14565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:30:37.023553 14565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:30:37.037985 14575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:37.038268 14577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:37.038076 14574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:30:37.038722 14565 server_base.cc:1034] running on GCE node
I20260810 23:30:37.039459 14565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:30:37.040618 14565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:30:37.041931 14565 hybrid_clock.cc:648] HybridClock initialized: now 1786404637041851 us; error 51 us; skew 500 ppm
I20260810 23:30:37.042313 14565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:30:37.044941 14565 webserver.cc:460] Webserver started at http://127.14.175.65:46335/ using document root <none> and password file <none>
I20260810 23:30:37.045774 14565 fs_manager.cc:359] Metadata directory not provided
I20260810 23:30:37.045893 14565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:30:37.046221 14565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:30:37.049508 14565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-0/data/instance:
uuid: "d2bdc296fc804a18b53ba07ed62e260a"
format_stamp: "Formatted at 2026-08-10 23:30:37 on dist-test-slave-rmhg"
server_key: "96d077a0fc26124b195df2b36bf344fb"
server_key_iv: "d2732d782f410e761d6e1c82e5f38cc9"
server_key_version: "encryptionkey@0"
I20260810 23:30:37.050402 14565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance:
uuid: "d2bdc296fc804a18b53ba07ed62e260a"
format_stamp: "Formatted at 2026-08-10 23:30:37 on dist-test-slave-rmhg"
server_key: "96d077a0fc26124b195df2b36bf344fb"
server_key_iv: "d2732d782f410e761d6e1c82e5f38cc9"
server_key_version: "encryptionkey@0"
I20260810 23:30:37.056164 14565 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:30:37.060132 14585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:37.061861 14565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:30:37.062054 14565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-0/wal
uuid: "d2bdc296fc804a18b53ba07ed62e260a"
format_stamp: "Formatted at 2026-08-10 23:30:37 on dist-test-slave-rmhg"
server_key: "96d077a0fc26124b195df2b36bf344fb"
server_key_iv: "d2732d782f410e761d6e1c82e5f38cc9"
server_key_version: "encryptionkey@0"
I20260810 23:30:37.062300 14565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:30:37.076463 14565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:30:37.077548 14565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:30:37.077865 14565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:30:37.078954 14565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:30:37.080642 14565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:30:37.080745 14565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:37.080862 14565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:30:37.080935 14565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:37.113023 14565 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.65:32811
I20260810 23:30:37.113111 14771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.65:32811 every 8 connection(s)
I20260810 23:30:37.114425 14565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-0/data/info.pb
I20260810 23:30:37.123579 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 14565
I20260810 23:30:37.123767 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-0/wal/instance
I20260810 23:30:37.124213 15037 external_mini_cluster.cc:1428] Setting key bcfa5d8ad60c38613377d89941d96ed1
I20260810 23:30:37.127326 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.66:0
--local_ip_for_outbound_sockets=127.14.175.66
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:35275
--builtin_ntp_servers=127.14.175.84:46721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:30:37.127887 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:37.117137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.65:40521 (local address 127.14.175.126:35275)
0810 23:30:37.117631 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:37.117636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:37.118930 (+ 1294us) server_negotiation.cc:407] Connection header received
0810 23:30:37.120801 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:37.120808 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:37.120892 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:37.121015 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:37.122285 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:37.123020 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:37.124827 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:37.125352 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:37.125506 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:37.126975 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:37.126981 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:37.126986 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:37.127008 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:37.127066 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:37.127084 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:37.127189 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:37.127437 (+ 248us) server_negotiation.cc:299] Negotiation successful
0810 23:30:37.127550 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":238,"threads_started":1}
I20260810 23:30:37.129191 14776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:37.117770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:35275 (local address 127.14.175.65:40521)
0810 23:30:37.118755 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:37.118795 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:37.120494 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:37.121345 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:37.121355 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:37.121456 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:30:37.122093 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:37.122112 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:37.123218 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:37.123222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:37.124656 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:37.124673 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:37.125580 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:37.125586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:37.125811 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:37.126615 (+ 804us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:30:37.126618 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:30:37.126664 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:30:37.126757 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:30:37.127196 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:30:37.127206 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:30:37.127213 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 23:30:37.127372 (+ 159us) client_negotiation.cc:240] Negotiation successful
0810 23:30:37.127923 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":316,"threads_started":1}
I20260810 23:30:37.130963 14773 heartbeater.cc:347] Connected to a master server at 127.14.175.126:35275
I20260810 23:30:37.131307 14773 heartbeater.cc:464] Registering TS with master...
I20260810 23:30:37.132157 14773 heartbeater.cc:511] Master 127.14.175.126:35275 requested a full tablet report, sending...
I20260810 23:30:37.134475 14485 ts_manager.cc:194] Registered new tserver with Master: d2bdc296fc804a18b53ba07ed62e260a (127.14.175.65:32811)
I20260810 23:30:37.136044 14485 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.175.65:40521
W20260810 23:30:37.481799 14778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:30:37.482136 14778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:30:37.482275 14778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:30:37.489682 14778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:30:37.489807 14778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:30:37.489961 14778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.66
I20260810 23:30:37.499658 14778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:46721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.175.66
--webserver_port=0
--tserver_master_addrs=127.14.175.126:35275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.66
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:30:37.501127 14778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:30:37.503098 14778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:30:37.517244 14791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:37.517232 14789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:37.517926 14787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:30:37.518231 14778 server_base.cc:1034] running on GCE node
I20260810 23:30:37.518896 14778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:30:37.519933 14778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:30:37.521190 14778 hybrid_clock.cc:648] HybridClock initialized: now 1786404637521107 us; error 48 us; skew 500 ppm
I20260810 23:30:37.521535 14778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:30:37.523885 14778 webserver.cc:460] Webserver started at http://127.14.175.66:36719/ using document root <none> and password file <none>
I20260810 23:30:37.524623 14778 fs_manager.cc:359] Metadata directory not provided
I20260810 23:30:37.524734 14778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:30:37.525033 14778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:30:37.528127 14778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-1/data/instance:
uuid: "2f0ba2d879b94f368c5131b0139a3078"
format_stamp: "Formatted at 2026-08-10 23:30:37 on dist-test-slave-rmhg"
server_key: "56a99cd255abe13cbce6be35dfb3cc07"
server_key_iv: "4f3e0631564a702e59a127d35bbe4ceb"
server_key_version: "encryptionkey@0"
I20260810 23:30:37.529011 14778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance:
uuid: "2f0ba2d879b94f368c5131b0139a3078"
format_stamp: "Formatted at 2026-08-10 23:30:37 on dist-test-slave-rmhg"
server_key: "56a99cd255abe13cbce6be35dfb3cc07"
server_key_iv: "4f3e0631564a702e59a127d35bbe4ceb"
server_key_version: "encryptionkey@0"
I20260810 23:30:37.536152 14778 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260810 23:30:37.540987 14799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:37.542999 14778 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:30:37.543228 14778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-1/wal
uuid: "2f0ba2d879b94f368c5131b0139a3078"
format_stamp: "Formatted at 2026-08-10 23:30:37 on dist-test-slave-rmhg"
server_key: "56a99cd255abe13cbce6be35dfb3cc07"
server_key_iv: "4f3e0631564a702e59a127d35bbe4ceb"
server_key_version: "encryptionkey@0"
I20260810 23:30:37.543498 14778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:30:37.561966 14778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:30:37.562858 14778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:30:37.563138 14778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:30:37.564177 14778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:30:37.565891 14778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:30:37.566006 14778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:37.566128 14778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:30:37.566221 14778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:37.598841 14778 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.66:36921
I20260810 23:30:37.598893 14974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.66:36921 every 8 connection(s)
I20260810 23:30:37.600255 14778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-1/data/info.pb
I20260810 23:30:37.601006 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 14778
I20260810 23:30:37.601235 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-1/wal/instance
I20260810 23:30:37.601818 15037 external_mini_cluster.cc:1428] Setting key 7c83b6f87f81cb1696cc941ff599e62d
I20260810 23:30:37.605029 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.175.67:0
--local_ip_for_outbound_sockets=127.14.175.67
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:35275
--builtin_ntp_servers=127.14.175.84:46721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:30:37.614132 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:37.603229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:45473 (local address 127.14.175.126:35275)
0810 23:30:37.603396 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:37.603400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:37.605351 (+ 1951us) server_negotiation.cc:407] Connection header received
0810 23:30:37.607292 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:37.607298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:37.607382 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:37.607485 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:37.608623 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:37.609625 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:37.611253 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:37.611788 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:37.611913 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:37.613365 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:37.613371 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:37.613375 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:37.613398 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:37.613437 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:37.613452 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:37.613531 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:37.613726 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 23:30:37.613828 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 23:30:37.615257 14979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:37.603941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:35275 (local address 127.14.175.66:45473)
0810 23:30:37.605176 (+ 1235us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:37.605227 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:37.606980 (+ 1753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:37.607694 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:37.607703 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:37.607800 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:30:37.608442 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:37.608461 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:37.609779 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:37.609783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:37.611076 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:37.611111 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:37.612006 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:37.612011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:37.612251 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:37.612960 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:30:37.612963 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:30:37.613018 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:30:37.613119 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:30:37.613540 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:30:37.613547 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:30:37.613553 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 23:30:37.613698 (+ 145us) client_negotiation.cc:240] Negotiation successful
0810 23:30:37.614033 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":340,"threads_started":1}
I20260810 23:30:37.616823 14975 heartbeater.cc:347] Connected to a master server at 127.14.175.126:35275
I20260810 23:30:37.617218 14975 heartbeater.cc:464] Registering TS with master...
I20260810 23:30:37.618119 14975 heartbeater.cc:511] Master 127.14.175.126:35275 requested a full tablet report, sending...
I20260810 23:30:37.619798 14485 ts_manager.cc:194] Registered new tserver with Master: 2f0ba2d879b94f368c5131b0139a3078 (127.14.175.66:36921)
I20260810 23:30:37.620880 14485 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.175.66:45473
W20260810 23:30:37.958220 14981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:30:37.958570 14981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:30:37.958706 14981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:30:37.966228 14981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:30:37.966355 14981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:30:37.966511 14981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.175.67
I20260810 23:30:37.976094 14981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:46721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.175.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.175.67
--webserver_port=0
--tserver_master_addrs=127.14.175.126:35275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.175.67
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:30:37.977646 14981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:30:37.979571 14981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:30:37.992913 14989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:37.992995 14992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:37.993011 14988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:30:37.993582 14981 server_base.cc:1034] running on GCE node
I20260810 23:30:37.994262 14981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:30:37.995246 14981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:30:37.996482 14981 hybrid_clock.cc:648] HybridClock initialized: now 1786404637996423 us; error 41 us; skew 500 ppm
I20260810 23:30:37.996827 14981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:30:37.999218 14981 webserver.cc:460] Webserver started at http://127.14.175.67:46241/ using document root <none> and password file <none>
I20260810 23:30:37.999953 14981 fs_manager.cc:359] Metadata directory not provided
I20260810 23:30:38.000077 14981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:30:38.000402 14981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:30:38.003556 14981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-2/data/instance:
uuid: "e0504c3e0081400dbb093cec18ad6781"
format_stamp: "Formatted at 2026-08-10 23:30:37 on dist-test-slave-rmhg"
server_key: "68abfe82b5eeb635b62d55fee79c3791"
server_key_iv: "20a06799d5d6c02479ae50ee0c686f53"
server_key_version: "encryptionkey@0"
I20260810 23:30:38.004367 14981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance:
uuid: "e0504c3e0081400dbb093cec18ad6781"
format_stamp: "Formatted at 2026-08-10 23:30:37 on dist-test-slave-rmhg"
server_key: "68abfe82b5eeb635b62d55fee79c3791"
server_key_iv: "20a06799d5d6c02479ae50ee0c686f53"
server_key_version: "encryptionkey@0"
I20260810 23:30:38.010149 14981 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260810 23:30:38.013967 15003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:38.015489 14981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:30:38.015688 14981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-2/wal
uuid: "e0504c3e0081400dbb093cec18ad6781"
format_stamp: "Formatted at 2026-08-10 23:30:37 on dist-test-slave-rmhg"
server_key: "68abfe82b5eeb635b62d55fee79c3791"
server_key_iv: "20a06799d5d6c02479ae50ee0c686f53"
server_key_version: "encryptionkey@0"
I20260810 23:30:38.015915 14981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:30:38.038410 14981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:30:38.039343 14981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:30:38.039660 14981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:30:38.040875 14981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:30:38.042582 14981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:30:38.042686 14981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:38.042797 14981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:30:38.042867 14981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:38.075675 14981 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.67:37621
I20260810 23:30:38.075786 15180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.67:37621 every 8 connection(s)
I20260810 23:30:38.077239 14981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-2/data/info.pb
I20260810 23:30:38.078894 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 14981
I20260810 23:30:38.079072 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-2/wal/instance
I20260810 23:30:38.079528 15037 external_mini_cluster.cc:1428] Setting key 4281d4a89fc49c1f9c077fd4cdb61dbb
I20260810 23:30:38.090016 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:38.080003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.67:50987 (local address 127.14.175.126:35275)
0810 23:30:38.080186 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:38.080193 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:38.081927 (+ 1734us) server_negotiation.cc:407] Connection header received
0810 23:30:38.083743 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:38.083747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:38.083816 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:38.083900 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:38.084965 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:38.085781 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:38.087369 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:38.087787 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:38.087885 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:38.089251 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:38.089256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:38.089260 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:38.089280 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:38.089312 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:38.089325 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:38.089393 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:38.089661 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 23:30:38.089745 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:30:38.091027 15184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:38.080619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:35275 (local address 127.14.175.67:50987)
0810 23:30:38.081762 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:38.081811 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:38.083477 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:38.084106 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:38.084114 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:38.084202 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:30:38.084792 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:38.084809 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:38.085938 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:38.085942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:38.087233 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:38.087246 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:38.087963 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:38.087967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:38.088199 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:38.088868 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:30:38.088871 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:30:38.088913 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:30:38.089018 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:30:38.089438 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:30:38.089447 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:30:38.089454 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 23:30:38.089619 (+ 165us) client_negotiation.cc:240] Negotiation successful
0810 23:30:38.089940 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":337,"threads_started":1}
I20260810 23:30:38.092397 15182 heartbeater.cc:347] Connected to a master server at 127.14.175.126:35275
I20260810 23:30:38.092720 15182 heartbeater.cc:464] Registering TS with master...
I20260810 23:30:38.093597 15182 heartbeater.cc:511] Master 127.14.175.126:35275 requested a full tablet report, sending...
I20260810 23:30:38.094972 14485 ts_manager.cc:194] Registered new tserver with Master: e0504c3e0081400dbb093cec18ad6781 (127.14.175.67:37621)
I20260810 23:30:38.095784 14485 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.175.67:50987
I20260810 23:30:38.107721 15037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:30:38.118376 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:38.112161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55438 (local address 127.14.175.126:35275)
0810 23:30:38.112336 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:38.112342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:38.112891 (+ 549us) server_negotiation.cc:407] Connection header received
0810 23:30:38.113072 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:38.113077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:38.113172 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:38.113261 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:38.113770 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:38.114466 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:38.116012 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:38.116444 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:38.116559 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:38.117685 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:38.117690 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:38.117693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:38.117712 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:38.117745 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:38.117758 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:38.117833 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:38.117969 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 23:30:38.118081 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:30:38.122377 14485 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:30:38.129565 14485 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:30:38.139482 14773 heartbeater.cc:503] Master 127.14.175.126:35275 was elected leader, sending a full tablet report...
I20260810 23:30:38.154356 15197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:38.143638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.66:36921 (local address 127.0.0.1:53680)
0810 23:30:38.145535 (+ 1897us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:38.145583 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:38.145783 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:38.146689 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:38.146696 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:38.146726 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:30:38.147030 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:38.147043 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:38.149015 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:38.149020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:38.151264 (+ 2244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:38.151278 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:38.153297 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:38.153304 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:38.153528 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:38.153581 (+ 53us) client_negotiation.cc:769] Sending connection context
0810 23:30:38.153705 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 23:30:38.153925 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1667,"mutex_wait_us":76,"thread_start_us":1541,"threads_started":1}
I20260810 23:30:38.154778 15203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:38.144493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60600 (local address 127.14.175.65:32811)
0810 23:30:38.145100 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:38.145109 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:38.146506 (+ 1397us) server_negotiation.cc:407] Connection header received
0810 23:30:38.146672 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:38.146680 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:38.146919 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:38.147120 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:30:38.148095 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:38.149714 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:38.152231 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:38.153619 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:38.153761 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:38.154064 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:38.154217 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 23:30:38.154396 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"spinlock_wait_cycles":24832,"thread_start_us":359,"threads_started":1}
I20260810 23:30:38.155057 15200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:38.144086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:32811 (local address 127.0.0.1:60600)
0810 23:30:38.146356 (+ 2270us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:38.146375 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:38.146509 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:38.147394 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:38.147398 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:38.147424 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:30:38.147910 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:38.147928 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:38.149837 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:38.149842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:38.151981 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:38.151994 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:38.153842 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:38.153848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:38.154030 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:38.154063 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 23:30:38.154166 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 23:30:38.154311 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2140,"thread_start_us":1984,"threads_started":1}
I20260810 23:30:38.155562 15198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:38.143821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53680 (local address 127.14.175.66:36921)
0810 23:30:38.144492 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:38.144503 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:38.145705 (+ 1202us) server_negotiation.cc:407] Connection header received
0810 23:30:38.146011 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:38.146020 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:38.146340 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:38.146551 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:30:38.147249 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:38.148801 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:38.151467 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:38.152927 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:38.153059 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:38.154613 (+ 1554us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:38.154696 (+ 83us) server_negotiation.cc:299] Negotiation successful
0810 23:30:38.155217 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":835,"threads_started":1}
I20260810 23:30:38.156782 15206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:38.147523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:37621 (local address 127.0.0.1:41286)
0810 23:30:38.148128 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:38.148156 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:38.148281 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:38.149268 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:38.149272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:38.149301 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:30:38.149804 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:38.149818 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:38.151771 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:38.151777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:38.153794 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:38.153807 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:38.156036 (+ 2229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:38.156043 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:38.156221 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:38.156253 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 23:30:38.156349 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 23:30:38.156488 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":328,"threads_started":1}
I20260810 23:30:38.157780 15204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:38.146862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41286 (local address 127.14.175.67:37621)
0810 23:30:38.147533 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:38.147541 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:38.148266 (+ 725us) server_negotiation.cc:407] Connection header received
0810 23:30:38.148466 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:38.148473 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:38.148714 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:38.148879 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:30:38.150033 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:38.151623 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:38.154033 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:38.155478 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:38.155593 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:38.155861 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:38.157265 (+ 1404us) server_negotiation.cc:299] Negotiation successful
0810 23:30:38.157454 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":332,"threads_started":1}
I20260810 23:30:38.159597 14663 tablet_service.cc:1467] Processing CreateTablet for tablet f852ea17a86b44d28bf5f7c30c2c8d36 (DEFAULT_TABLE table=test-table [id=84f61cc2517d4b7582fdcb0ac5279b3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:30:38.159597 14875 tablet_service.cc:1467] Processing CreateTablet for tablet f852ea17a86b44d28bf5f7c30c2c8d36 (DEFAULT_TABLE table=test-table [id=84f61cc2517d4b7582fdcb0ac5279b3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:30:38.161265 14663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f852ea17a86b44d28bf5f7c30c2c8d36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:30:38.161266 14875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f852ea17a86b44d28bf5f7c30c2c8d36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:30:38.163728 15081 tablet_service.cc:1467] Processing CreateTablet for tablet f852ea17a86b44d28bf5f7c30c2c8d36 (DEFAULT_TABLE table=test-table [id=84f61cc2517d4b7582fdcb0ac5279b3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:30:38.165330 15081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f852ea17a86b44d28bf5f7c30c2c8d36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:30:38.176823 15207 tablet_bootstrap.cc:492] T f852ea17a86b44d28bf5f7c30c2c8d36 P 2f0ba2d879b94f368c5131b0139a3078: Bootstrap starting.
I20260810 23:30:38.178406 15211 tablet_bootstrap.cc:492] T f852ea17a86b44d28bf5f7c30c2c8d36 P d2bdc296fc804a18b53ba07ed62e260a: Bootstrap starting.
I20260810 23:30:38.179497 15212 tablet_bootstrap.cc:492] T f852ea17a86b44d28bf5f7c30c2c8d36 P e0504c3e0081400dbb093cec18ad6781: Bootstrap starting.
I20260810 23:30:38.182615 15211 tablet_bootstrap.cc:654] T f852ea17a86b44d28bf5f7c30c2c8d36 P d2bdc296fc804a18b53ba07ed62e260a: Neither blocks nor log segments found. Creating new log.
I20260810 23:30:38.183115 15212 tablet_bootstrap.cc:654] T f852ea17a86b44d28bf5f7c30c2c8d36 P e0504c3e0081400dbb093cec18ad6781: Neither blocks nor log segments found. Creating new log.
I20260810 23:30:38.184085 15212 log.cc:824] T f852ea17a86b44d28bf5f7c30c2c8d36 P e0504c3e0081400dbb093cec18ad6781: Log is configured to *not* fsync() on all Append() calls
I20260810 23:30:38.184084 15211 log.cc:824] T f852ea17a86b44d28bf5f7c30c2c8d36 P d2bdc296fc804a18b53ba07ed62e260a: Log is configured to *not* fsync() on all Append() calls
I20260810 23:30:38.184569 15207 tablet_bootstrap.cc:654] T f852ea17a86b44d28bf5f7c30c2c8d36 P 2f0ba2d879b94f368c5131b0139a3078: Neither blocks nor log segments found. Creating new log.
I20260810 23:30:38.185815 15207 log.cc:824] T f852ea17a86b44d28bf5f7c30c2c8d36 P 2f0ba2d879b94f368c5131b0139a3078: Log is configured to *not* fsync() on all Append() calls
I20260810 23:30:38.187199 15212 tablet_bootstrap.cc:492] T f852ea17a86b44d28bf5f7c30c2c8d36 P e0504c3e0081400dbb093cec18ad6781: No bootstrap required, opened a new log
I20260810 23:30:38.187503 15212 ts_tablet_manager.cc:1397] T f852ea17a86b44d28bf5f7c30c2c8d36 P e0504c3e0081400dbb093cec18ad6781: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.008s
I20260810 23:30:38.188982 15207 tablet_bootstrap.cc:492] T f852ea17a86b44d28bf5f7c30c2c8d36 P 2f0ba2d879b94f368c5131b0139a3078: No bootstrap required, opened a new log
I20260810 23:30:38.189307 15207 ts_tablet_manager.cc:1397] T f852ea17a86b44d28bf5f7c30c2c8d36 P 2f0ba2d879b94f368c5131b0139a3078: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.003s
I20260810 23:30:38.189453 15211 tablet_bootstrap.cc:492] T f852ea17a86b44d28bf5f7c30c2c8d36 P d2bdc296fc804a18b53ba07ed62e260a: No bootstrap required, opened a new log
I20260810 23:30:38.189757 15211 ts_tablet_manager.cc:1397] T f852ea17a86b44d28bf5f7c30c2c8d36 P d2bdc296fc804a18b53ba07ed62e260a: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20260810 23:30:38.195142 15212 raft_consensus.cc:348] T f852ea17a86b44d28bf5f7c30c2c8d36 P e0504c3e0081400dbb093cec18ad6781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0ba2d879b94f368c5131b0139a3078" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36921 } } peers { permanent_uuid: "e0504c3e0081400dbb093cec18ad6781" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37621 } } peers { permanent_uuid: "d2bdc296fc804a18b53ba07ed62e260a" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 32811 } }
I20260810 23:30:38.195544 15212 raft_consensus.cc:374] T f852ea17a86b44d28bf5f7c30c2c8d36 P e0504c3e0081400dbb093cec18ad6781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:30:38.195650 15212 raft_consensus.cc:729] T f852ea17a86b44d28bf5f7c30c2c8d36 P e0504c3e0081400dbb093cec18ad6781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0504c3e0081400dbb093cec18ad6781, State: Initialized, Role: FOLLOWER
I20260810 23:30:38.196326 15212 consensus_queue.cc:260] T f852ea17a86b44d28bf5f7c30c2c8d36 P e0504c3e0081400dbb093cec18ad6781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0ba2d879b94f368c5131b0139a3078" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36921 } } peers { permanent_uuid: "e0504c3e0081400dbb093cec18ad6781" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37621 } } peers { permanent_uuid: "d2bdc296fc804a18b53ba07ed62e260a" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 32811 } }
I20260810 23:30:38.197494 15182 heartbeater.cc:503] Master 127.14.175.126:35275 was elected leader, sending a full tablet report...
I20260810 23:30:38.198088 15212 ts_tablet_manager.cc:1428] T f852ea17a86b44d28bf5f7c30c2c8d36 P e0504c3e0081400dbb093cec18ad6781: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260810 23:30:38.199942 15207 raft_consensus.cc:348] T f852ea17a86b44d28bf5f7c30c2c8d36 P 2f0ba2d879b94f368c5131b0139a3078 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0ba2d879b94f368c5131b0139a3078" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36921 } } peers { permanent_uuid: "e0504c3e0081400dbb093cec18ad6781" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37621 } } peers { permanent_uuid: "d2bdc296fc804a18b53ba07ed62e260a" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 32811 } }
I20260810 23:30:38.200435 15207 raft_consensus.cc:374] T f852ea17a86b44d28bf5f7c30c2c8d36 P 2f0ba2d879b94f368c5131b0139a3078 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:30:38.200542 15207 raft_consensus.cc:729] T f852ea17a86b44d28bf5f7c30c2c8d36 P 2f0ba2d879b94f368c5131b0139a3078 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f0ba2d879b94f368c5131b0139a3078, State: Initialized, Role: FOLLOWER
I20260810 23:30:38.201231 15207 consensus_queue.cc:260] T f852ea17a86b44d28bf5f7c30c2c8d36 P 2f0ba2d879b94f368c5131b0139a3078 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0ba2d879b94f368c5131b0139a3078" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36921 } } peers { permanent_uuid: "e0504c3e0081400dbb093cec18ad6781" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37621 } } peers { permanent_uuid: "d2bdc296fc804a18b53ba07ed62e260a" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 32811 } }
I20260810 23:30:38.201205 15211 raft_consensus.cc:348] T f852ea17a86b44d28bf5f7c30c2c8d36 P d2bdc296fc804a18b53ba07ed62e260a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0ba2d879b94f368c5131b0139a3078" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36921 } } peers { permanent_uuid: "e0504c3e0081400dbb093cec18ad6781" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37621 } } peers { permanent_uuid: "d2bdc296fc804a18b53ba07ed62e260a" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 32811 } }
I20260810 23:30:38.201776 15211 raft_consensus.cc:374] T f852ea17a86b44d28bf5f7c30c2c8d36 P d2bdc296fc804a18b53ba07ed62e260a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:30:38.201889 15211 raft_consensus.cc:729] T f852ea17a86b44d28bf5f7c30c2c8d36 P d2bdc296fc804a18b53ba07ed62e260a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2bdc296fc804a18b53ba07ed62e260a, State: Initialized, Role: FOLLOWER
I20260810 23:30:38.202591 14975 heartbeater.cc:503] Master 127.14.175.126:35275 was elected leader, sending a full tablet report...
I20260810 23:30:38.203317 15207 ts_tablet_manager.cc:1428] T f852ea17a86b44d28bf5f7c30c2c8d36 P 2f0ba2d879b94f368c5131b0139a3078: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260810 23:30:38.203917 15211 consensus_queue.cc:260] T f852ea17a86b44d28bf5f7c30c2c8d36 P d2bdc296fc804a18b53ba07ed62e260a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0ba2d879b94f368c5131b0139a3078" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36921 } } peers { permanent_uuid: "e0504c3e0081400dbb093cec18ad6781" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37621 } } peers { permanent_uuid: "d2bdc296fc804a18b53ba07ed62e260a" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 32811 } }
I20260810 23:30:38.205788 15211 ts_tablet_manager.cc:1428] T f852ea17a86b44d28bf5f7c30c2c8d36 P d2bdc296fc804a18b53ba07ed62e260a: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
W20260810 23:30:38.328984 15183 tablet.cc:2369] T f852ea17a86b44d28bf5f7c30c2c8d36 P e0504c3e0081400dbb093cec18ad6781: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:30:38.332916 15221 raft_consensus.cc:482] T f852ea17a86b44d28bf5f7c30c2c8d36 P 2f0ba2d879b94f368c5131b0139a3078 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:30:38.333346 15221 raft_consensus.cc:504] T f852ea17a86b44d28bf5f7c30c2c8d36 P 2f0ba2d879b94f368c5131b0139a3078 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0ba2d879b94f368c5131b0139a3078" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36921 } } peers { permanent_uuid: "e0504c3e0081400dbb093cec18ad6781" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37621 } } peers { permanent_uuid: "d2bdc296fc804a18b53ba07ed62e260a" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 32811 } }
I20260810 23:30:38.335179 15221 leader_election.cc:288] T f852ea17a86b44d28bf5f7c30c2c8d36 P 2f0ba2d879b94f368c5131b0139a3078 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0504c3e0081400dbb093cec18ad6781 (127.14.175.67:37621), d2bdc296fc804a18b53ba07ed62e260a (127.14.175.65:32811)
I20260810 23:30:38.343369 15204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:38.335659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:52791 (local address 127.14.175.67:37621)
0810 23:30:38.335866 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:38.335871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:38.336697 (+ 826us) server_negotiation.cc:407] Connection header received
0810 23:30:38.336906 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:38.336911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:38.336988 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:38.337092 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:30:38.337893 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:38.339005 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:38.341568 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:38.342381 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:38.342560 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:38.342636 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:38.342992 (+ 356us) server_negotiation.cc:299] Negotiation successful
0810 23:30:38.343113 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 23:30:38.343369 15203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:38.335654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.175.66:59055 (local address 127.14.175.65:32811)
0810 23:30:38.335857 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:38.335864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:38.336714 (+ 850us) server_negotiation.cc:407] Connection header received
0810 23:30:38.336887 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:38.336892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:38.336989 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:38.337093 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:30:38.337893 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:38.339033 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:38.341588 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:38.342416 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:38.342585 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:38.342655 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:38.342982 (+ 327us) server_negotiation.cc:299] Negotiation successful
0810 23:30:38.343125 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 23:30:38.343801 15223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:38.335588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:32811 (local address 127.14.175.66:59055)
0810 23:30:38.336530 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:38.336557 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:38.336747 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:38.337120 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:38.337125 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:38.337205 (+ 80us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:30:38.337729 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:38.337745 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:38.339203 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:38.339208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:38.341337 (+ 2129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:38.341357 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:38.342629 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:38.342636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:38.342819 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:38.342846 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 23:30:38.342924 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 23:30:38.343062 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"mutex_wait_us":92,"spinlock_wait_cycles":3968,"thread_start_us":501,"threads_started":1}
I20260810 23:30:38.343847 15224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:38.335673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.67:37621 (local address 127.14.175.66:52791)
0810 23:30:38.336570 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:38.336583 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:30:38.336791 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:30:38.337124 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:30:38.337128 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:30:38.337193 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:30:38.337740 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:38.337749 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:38.339185 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:38.339190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:38.341338 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:30:38.341357 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:30:38.342629 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:30:38.342637 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:30:38.342834 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:38.342859 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 23:30:38.342944 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 23:30:38.343072 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"mutex_wait_us":169,"spinlock_wait_cycles":45184,"thread_start_us":401,"threads_started":1}
I20260810 23:30:38.344879 15114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f852ea17a86b44d28bf5f7c30c2c8d36" candidate_uuid: "2f0ba2d879b94f368c5131b0139a3078" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0504c3e0081400dbb093cec18ad6781" is_pre_election: true
I20260810 23:30:38.344882 14701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f852ea17a86b44d28bf5f7c30c2c8d36" candidate_uuid: "2f0ba2d879b94f368c5131b0139a3078" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2bdc296fc804a18b53ba07ed62e260a" is_pre_election: true
I20260810 23:30:38.345414 15114 raft_consensus.cc:2454] T f852ea17a86b44d28bf5f7c30c2c8d36 P e0504c3e0081400dbb093cec18ad6781 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f0ba2d879b94f368c5131b0139a3078 in term 0.
I20260810 23:30:38.345424 14701 raft_consensus.cc:2454] T f852ea17a86b44d28bf5f7c30c2c8d36 P d2bdc296fc804a18b53ba07ed62e260a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f0ba2d879b94f368c5131b0139a3078 in term 0.
I20260810 23:30:38.346081 14804 leader_election.cc:302] T f852ea17a86b44d28bf5f7c30c2c8d36 P 2f0ba2d879b94f368c5131b0139a3078 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f0ba2d879b94f368c5131b0139a3078, d2bdc296fc804a18b53ba07ed62e260a; no voters:
I20260810 23:30:38.346567 15221 raft_consensus.cc:2781] T f852ea17a86b44d28bf5f7c30c2c8d36 P 2f0ba2d879b94f368c5131b0139a3078 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:30:38.346706 15221 raft_consensus.cc:482] T f852ea17a86b44d28bf5f7c30c2c8d36 P 2f0ba2d879b94f368c5131b0139a3078 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:30:38.346810 15221 raft_consensus.cc:3037] T f852ea17a86b44d28bf5f7c30c2c8d36 P 2f0ba2d879b94f368c5131b0139a3078 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:30:38.349138 15221 raft_consensus.cc:504] T f852ea17a86b44d28bf5f7c30c2c8d36 P 2f0ba2d879b94f368c5131b0139a3078 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0ba2d879b94f368c5131b0139a3078" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36921 } } peers { permanent_uuid: "e0504c3e0081400dbb093cec18ad6781" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37621 } } peers { permanent_uuid: "d2bdc296fc804a18b53ba07ed62e260a" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 32811 } }
I20260810 23:30:38.349872 15221 leader_election.cc:288] T f852ea17a86b44d28bf5f7c30c2c8d36 P 2f0ba2d879b94f368c5131b0139a3078 [CANDIDATE]: Term 1 election: Requested vote from peers e0504c3e0081400dbb093cec18ad6781 (127.14.175.67:37621), d2bdc296fc804a18b53ba07ed62e260a (127.14.175.65:32811)
I20260810 23:30:38.350368 15114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f852ea17a86b44d28bf5f7c30c2c8d36" candidate_uuid: "2f0ba2d879b94f368c5131b0139a3078" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0504c3e0081400dbb093cec18ad6781"
I20260810 23:30:38.350392 14701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f852ea17a86b44d28bf5f7c30c2c8d36" candidate_uuid: "2f0ba2d879b94f368c5131b0139a3078" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2bdc296fc804a18b53ba07ed62e260a"
I20260810 23:30:38.350589 15114 raft_consensus.cc:3037] T f852ea17a86b44d28bf5f7c30c2c8d36 P e0504c3e0081400dbb093cec18ad6781 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:30:38.350581 14701 raft_consensus.cc:3037] T f852ea17a86b44d28bf5f7c30c2c8d36 P d2bdc296fc804a18b53ba07ed62e260a [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:30:38.352571 14976 tablet.cc:2369] T f852ea17a86b44d28bf5f7c30c2c8d36 P 2f0ba2d879b94f368c5131b0139a3078: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:30:38.352888 14701 raft_consensus.cc:2454] T f852ea17a86b44d28bf5f7c30c2c8d36 P d2bdc296fc804a18b53ba07ed62e260a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f0ba2d879b94f368c5131b0139a3078 in term 1.
I20260810 23:30:38.353193 15114 raft_consensus.cc:2454] T f852ea17a86b44d28bf5f7c30c2c8d36 P e0504c3e0081400dbb093cec18ad6781 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f0ba2d879b94f368c5131b0139a3078 in term 1.
I20260810 23:30:38.353446 14804 leader_election.cc:302] T f852ea17a86b44d28bf5f7c30c2c8d36 P 2f0ba2d879b94f368c5131b0139a3078 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f0ba2d879b94f368c5131b0139a3078, d2bdc296fc804a18b53ba07ed62e260a; no voters:
I20260810 23:30:38.353756 15221 raft_consensus.cc:2781] T f852ea17a86b44d28bf5f7c30c2c8d36 P 2f0ba2d879b94f368c5131b0139a3078 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:30:38.354233 15221 raft_consensus.cc:686] T f852ea17a86b44d28bf5f7c30c2c8d36 P 2f0ba2d879b94f368c5131b0139a3078 [term 1 LEADER]: Becoming Leader. State: Replica: 2f0ba2d879b94f368c5131b0139a3078, State: Running, Role: LEADER
I20260810 23:30:38.354758 15221 consensus_queue.cc:237] T f852ea17a86b44d28bf5f7c30c2c8d36 P 2f0ba2d879b94f368c5131b0139a3078 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0ba2d879b94f368c5131b0139a3078" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36921 } } peers { permanent_uuid: "e0504c3e0081400dbb093cec18ad6781" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37621 } } peers { permanent_uuid: "d2bdc296fc804a18b53ba07ed62e260a" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 32811 } }
I20260810 23:30:38.359467 14485 catalog_manager.cc:5495] T f852ea17a86b44d28bf5f7c30c2c8d36 P 2f0ba2d879b94f368c5131b0139a3078 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f0ba2d879b94f368c5131b0139a3078 (127.14.175.66). New cstate: current_term: 1 leader_uuid: "2f0ba2d879b94f368c5131b0139a3078" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0ba2d879b94f368c5131b0139a3078" member_type: VOTER last_known_addr { host: "127.14.175.66" port: 36921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0504c3e0081400dbb093cec18ad6781" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2bdc296fc804a18b53ba07ed62e260a" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 32811 } health_report { overall_health: UNKNOWN } } }
W20260810 23:30:38.366976 14774 tablet.cc:2369] T f852ea17a86b44d28bf5f7c30c2c8d36 P d2bdc296fc804a18b53ba07ed62e260a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:30:38.398445 15198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:38.392966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53694 (local address 127.14.175.66:36921)
0810 23:30:38.393194 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:38.393200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:38.393299 (+ 99us) server_negotiation.cc:407] Connection header received
0810 23:30:38.393521 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:38.393525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:38.393643 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:38.393794 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:30:38.394170 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:38.394872 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:38.396238 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:38.396680 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:38.396789 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:38.397309 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:30:38.397848 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:30:38.397989 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:38.398089 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 23:30:38.398174 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 23:30:38.407246 15114 raft_consensus.cc:1261] T f852ea17a86b44d28bf5f7c30c2c8d36 P e0504c3e0081400dbb093cec18ad6781 [term 1 FOLLOWER]: Refusing update from remote peer 2f0ba2d879b94f368c5131b0139a3078: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:30:38.407246 14701 raft_consensus.cc:1261] T f852ea17a86b44d28bf5f7c30c2c8d36 P d2bdc296fc804a18b53ba07ed62e260a [term 1 FOLLOWER]: Refusing update from remote peer 2f0ba2d879b94f368c5131b0139a3078: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:30:38.408483 15225 consensus_queue.cc:1035] T f852ea17a86b44d28bf5f7c30c2c8d36 P 2f0ba2d879b94f368c5131b0139a3078 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2bdc296fc804a18b53ba07ed62e260a" member_type: VOTER last_known_addr { host: "127.14.175.65" port: 32811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:30:38.408906 15221 consensus_queue.cc:1035] T f852ea17a86b44d28bf5f7c30c2c8d36 P 2f0ba2d879b94f368c5131b0139a3078 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0504c3e0081400dbb093cec18ad6781" member_type: VOTER last_known_addr { host: "127.14.175.67" port: 37621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:30:38.420276 15229 mvcc.cc:205] Tried to move back new op lower bound from 7317113398904254464 to 7317113398704975872. Current Snapshot: MvccSnapshot[applied={T|T < 7317113398904254464}]
I20260810 23:30:38.435002 14485 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:30:38.435312 14485 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:30:38.439210 14485 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f62d50aa1324f15b10d0410c05a3053: Sending DeleteTablet for 3 replicas of tablet f852ea17a86b44d28bf5f7c30c2c8d36
I20260810 23:30:38.440604 14875 tablet_service.cc:1514] Processing DeleteTablet for tablet f852ea17a86b44d28bf5f7c30c2c8d36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:30:38 UTC) from {username='slave'} at 127.0.0.1:53680
I20260810 23:30:38.440609 14663 tablet_service.cc:1514] Processing DeleteTablet for tablet f852ea17a86b44d28bf5f7c30c2c8d36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:30:38 UTC) from {username='slave'} at 127.0.0.1:60600
I20260810 23:30:38.440835 15081 tablet_service.cc:1514] Processing DeleteTablet for tablet f852ea17a86b44d28bf5f7c30c2c8d36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:30:38 UTC) from {username='slave'} at 127.0.0.1:41286
I20260810 23:30:38.442643 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 14565
I20260810 23:30:38.449875 15244 tablet_replica.cc:330] T f852ea17a86b44d28bf5f7c30c2c8d36 P e0504c3e0081400dbb093cec18ad6781: stopping tablet replica
I20260810 23:30:38.451372 15244 raft_consensus.cc:2229] T f852ea17a86b44d28bf5f7c30c2c8d36 P e0504c3e0081400dbb093cec18ad6781 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:30:38.451977 15244 raft_consensus.cc:2258] T f852ea17a86b44d28bf5f7c30c2c8d36 P e0504c3e0081400dbb093cec18ad6781 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:30:38.455813 15244 ts_tablet_manager.cc:1905] T f852ea17a86b44d28bf5f7c30c2c8d36 P e0504c3e0081400dbb093cec18ad6781: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:30:38.464015 15243 tablet_replica.cc:330] T f852ea17a86b44d28bf5f7c30c2c8d36 P 2f0ba2d879b94f368c5131b0139a3078: stopping tablet replica
I20260810 23:30:38.466284 15244 ts_tablet_manager.cc:1918] T f852ea17a86b44d28bf5f7c30c2c8d36 P e0504c3e0081400dbb093cec18ad6781: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:30:38.466480 15244 log.cc:1196] T f852ea17a86b44d28bf5f7c30c2c8d36 P e0504c3e0081400dbb093cec18ad6781: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-2/wal/wals/f852ea17a86b44d28bf5f7c30c2c8d36
I20260810 23:30:38.466616 15203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:38.444237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60614 (local address 127.14.175.65:32811)
0810 23:30:38.444439 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:38.444447 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:38.457004 (+ 12557us) server_negotiation.cc:407] Connection header received
0810 23:30:38.457588 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:38.457595 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:38.457740 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:38.457834 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:38.458582 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:38.459924 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:38.461880 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:38.462518 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:38.462617 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:38.464669 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:38.464682 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:38.464702 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:38.465566 (+ 864us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:38.465645 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:38.465671 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:38.465777 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:38.466039 (+ 262us) server_negotiation.cc:299] Negotiation successful
0810 23:30:38.466174 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 23:30:38.467056 15244 ts_tablet_manager.cc:1939] T f852ea17a86b44d28bf5f7c30c2c8d36 P e0504c3e0081400dbb093cec18ad6781: Deleting consensus metadata
I20260810 23:30:38.468094 14766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:30:38.469129 14453 catalog_manager.cc:4841] TS e0504c3e0081400dbb093cec18ad6781 (127.14.175.67:37621): tablet f852ea17a86b44d28bf5f7c30c2c8d36 (table test-table [id=84f61cc2517d4b7582fdcb0ac5279b3c]) successfully deleted
I20260810 23:30:38.469795 15243 raft_consensus.cc:2229] T f852ea17a86b44d28bf5f7c30c2c8d36 P 2f0ba2d879b94f368c5131b0139a3078 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:30:38.470659 15243 raft_consensus.cc:2258] T f852ea17a86b44d28bf5f7c30c2c8d36 P 2f0ba2d879b94f368c5131b0139a3078 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:30:38.473804 15243 ts_tablet_manager.cc:1905] T f852ea17a86b44d28bf5f7c30c2c8d36 P 2f0ba2d879b94f368c5131b0139a3078: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:30:38.483114 15243 ts_tablet_manager.cc:1918] T f852ea17a86b44d28bf5f7c30c2c8d36 P 2f0ba2d879b94f368c5131b0139a3078: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:30:38.483361 15243 log.cc:1196] T f852ea17a86b44d28bf5f7c30c2c8d36 P 2f0ba2d879b94f368c5131b0139a3078: Deleting WAL directory at /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404299514076-15037-0/minicluster-data/ts-1/wal/wals/f852ea17a86b44d28bf5f7c30c2c8d36
I20260810 23:30:38.484052 15243 ts_tablet_manager.cc:1939] T f852ea17a86b44d28bf5f7c30c2c8d36 P 2f0ba2d879b94f368c5131b0139a3078: Deleting consensus metadata
I20260810 23:30:38.485805 14453 catalog_manager.cc:4841] TS 2f0ba2d879b94f368c5131b0139a3078 (127.14.175.66:36921): tablet f852ea17a86b44d28bf5f7c30c2c8d36 (table test-table [id=84f61cc2517d4b7582fdcb0ac5279b3c]) successfully deleted
I20260810 23:30:38.610608 15242 tablet_replica.cc:330] T f852ea17a86b44d28bf5f7c30c2c8d36 P d2bdc296fc804a18b53ba07ed62e260a: stopping tablet replica
I20260810 23:30:38.611114 15242 raft_consensus.cc:2229] T f852ea17a86b44d28bf5f7c30c2c8d36 P d2bdc296fc804a18b53ba07ed62e260a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:30:38.611514 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 14565
I20260810 23:30:38.611670 15242 raft_consensus.cc:2258] T f852ea17a86b44d28bf5f7c30c2c8d36 P d2bdc296fc804a18b53ba07ed62e260a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 23:30:38.645040 14458 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.175.65:32811 (error 108)
I20260810 23:30:38.645889 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 14778
I20260810 23:30:38.647164 15206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:38.646454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:32811 (local address 127.0.0.1:60630)
0810 23:30:38.646792 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:38.646966 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:32811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
W20260810 23:30:38.647468 14458 catalog_manager.cc:4584] TS d2bdc296fc804a18b53ba07ed62e260a (127.14.175.65:32811): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f852ea17a86b44d28bf5f7c30c2c8d36: Network error: Client connection negotiation failed: client connection to 127.14.175.65:32811: connect: Connection refused (error 111)
I20260810 23:30:38.653658 15198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:38.646799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53702 (local address 127.14.175.66:36921)
0810 23:30:38.646993 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:38.646999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:38.647023 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:30:38.647247 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:38.647253 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:38.647371 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:38.647499 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:38.647908 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:38.648838 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:38.650376 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:38.650848 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:38.650966 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:38.652103 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:38.652113 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:38.652131 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:38.652792 (+ 661us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:38.652843 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:38.652866 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:38.652978 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:38.653136 (+ 158us) server_negotiation.cc:299] Negotiation successful
0810 23:30:38.653275 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 23:30:38.654423 14961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:30:38.701328 15206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:38.700703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:32811 (local address 127.0.0.1:60644)
0810 23:30:38.700989 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:38.701108 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:32811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260810 23:30:38.702464 15206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:38.702073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:32811 (local address 127.0.0.1:60660)
0810 23:30:38.702220 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:38.702309 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:32811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 23:30:38.763191 15206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:38.762493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:32811 (local address 127.0.0.1:60662)
0810 23:30:38.762831 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:38.762955 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:32811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260810 23:30:38.764534 15206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:38.764086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:32811 (local address 127.0.0.1:60674)
0810 23:30:38.764267 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:38.764364 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:32811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 23:30:38.768780 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 14778
I20260810 23:30:38.794373 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 14981
I20260810 23:30:38.801440 15204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:38.795217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41292 (local address 127.14.175.67:37621)
0810 23:30:38.795413 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:38.795419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:38.795441 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:30:38.795607 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:38.795612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:38.795712 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:38.795800 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:38.796224 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:38.796950 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:38.798396 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:38.798813 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:38.798925 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:38.800001 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:38.800010 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:38.800028 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:38.800628 (+ 600us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:38.800683 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:38.800704 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:38.800814 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:38.800950 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 23:30:38.801095 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 23:30:38.802172 15176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:30:38.845424 15206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:38.844742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:32811 (local address 127.0.0.1:60678)
0810 23:30:38.845014 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:38.845193 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:32811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260810 23:30:38.846668 15206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:38.846174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.65:32811 (local address 127.0.0.1:60694)
0810 23:30:38.846411 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0810 23:30:38.846511 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.65:32811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260810 23:30:38.890786 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 14981
I20260810 23:30:38.910771 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 14424
I20260810 23:30:38.911989 14526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:30:39.023454 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 14424
2026-08-10T23:30:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3045 ms)
[----------] 6 tests from EncryptionPolicyTest (23895 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:30:39 dist-test-slave-rmhg krb5kdc[15258](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:30:39 dist-test-slave-rmhg krb5kdc[15258](info): set up 2 sockets
Aug 10 23:30:39 dist-test-slave-rmhg krb5kdc[15258](info): commencing operation
krb5kdc: starting...
W20260810 23:30:40.299930 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.224s user 0.000s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:30:40 dist-test-slave-rmhg krb5kdc[15258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:30:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:30:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:30:40.636282 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34141
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:30:40.999878 15275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:30:41.000228 15275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:30:41.000357 15275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:30:41.008050 15275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:30:41.008199 15275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:30:41.008282 15275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:30:41.008374 15275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:30:41.008431 15275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:30:41.018983 15275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34141
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:30:41.020435 15275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:30:41.021953 15275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:30:41.034245 15285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:41.034214 15290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:41.034288 15287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:30:41.035231 15275 server_base.cc:1034] running on GCE node
I20260810 23:30:41.036046 15275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:30:41.037581 15275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:30:41.038800 15275 hybrid_clock.cc:648] HybridClock initialized: now 1786404641038748 us; error 37 us; skew 500 ppm
I20260810 23:30:41.039170 15275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:30:41 dist-test-slave-rmhg krb5kdc[15258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:30:41.045998 15275 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:30:41.047838 15275 webserver.cc:460] Webserver started at http://127.0.0.1:34993/ using document root <none> and password file <none>
I20260810 23:30:41.048583 15275 fs_manager.cc:359] Metadata directory not provided
I20260810 23:30:41.048731 15275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:30:41.049073 15275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:30:41.052404 15275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "55fb2775f4d64632a2eed20429002df7"
format_stamp: "Formatted at 2026-08-10 23:30:41 on dist-test-slave-rmhg"
server_key: "0898c7346ad5bf254c43b255437b1d64"
server_key_iv: "d747099ae6b85a5df76212a73c08a9bf"
server_key_version: "encryptionkey@0"
I20260810 23:30:41.053330 15275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "55fb2775f4d64632a2eed20429002df7"
format_stamp: "Formatted at 2026-08-10 23:30:41 on dist-test-slave-rmhg"
server_key: "0898c7346ad5bf254c43b255437b1d64"
server_key_iv: "d747099ae6b85a5df76212a73c08a9bf"
server_key_version: "encryptionkey@0"
I20260810 23:30:41.059021 15275 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:30:41.062904 15302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:41.064324 15275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:30:41.064520 15275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "55fb2775f4d64632a2eed20429002df7"
format_stamp: "Formatted at 2026-08-10 23:30:41 on dist-test-slave-rmhg"
server_key: "0898c7346ad5bf254c43b255437b1d64"
server_key_iv: "d747099ae6b85a5df76212a73c08a9bf"
server_key_version: "encryptionkey@0"
I20260810 23:30:41.064764 15275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:30:41.119292 15275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:30:41.124135 15275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:30:41.124446 15275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:30:41.144845 15275 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34141
I20260810 23:30:41.144894 15384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34141 every 8 connection(s)
I20260810 23:30:41.146431 15275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:30:41.150693 15385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:30:41.152307 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 15275
I20260810 23:30:41.152540 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:30:41.153432 15037 external_mini_cluster.cc:1428] Setting key 22b2ed1e40ff950f6669987f6951374e
I20260810 23:30:41.166263 15385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55fb2775f4d64632a2eed20429002df7: Bootstrap starting.
Aug 10 23:30:41 dist-test-slave-rmhg krb5kdc[15258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 23:30:41.170825 15385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55fb2775f4d64632a2eed20429002df7: Neither blocks nor log segments found. Creating new log.
I20260810 23:30:41.172185 15385 log.cc:824] T 00000000000000000000000000000000 P 55fb2775f4d64632a2eed20429002df7: Log is configured to *not* fsync() on all Append() calls
I20260810 23:30:41.176363 15385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55fb2775f4d64632a2eed20429002df7: No bootstrap required, opened a new log
I20260810 23:30:41.180756 15389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:41.155574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36017 (local address 127.0.0.1:34141)
0810 23:30:41.156546 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:41.156559 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:41.156606 (+ 47us) server_negotiation.cc:407] Connection header received
0810 23:30:41.158351 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:41.158383 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:41.158866 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:41.159231 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:41.161296 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:41.162748 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:41.164819 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:41.165706 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:41.165870 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:41.171450 (+ 5580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:41.171460 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:41.171478 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:30:41.171528 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:41.175713 (+ 4185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:41.176766 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:41.176774 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:41.176782 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:41.176992 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:41.177587 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:41.177591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:41.177593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:41.178180 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:41.178557 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:41.179094 (+ 537us) server_negotiation.cc:299] Negotiation successful
0810 23:30:41.179514 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":447,"threads_started":1}
I20260810 23:30:41.185572 15037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:30:41.185575 15385 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55fb2775f4d64632a2eed20429002df7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fb2775f4d64632a2eed20429002df7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34141 } }
I20260810 23:30:41.186118 15385 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55fb2775f4d64632a2eed20429002df7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:30:41.186230 15385 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55fb2775f4d64632a2eed20429002df7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55fb2775f4d64632a2eed20429002df7, State: Initialized, Role: FOLLOWER
I20260810 23:30:41.186930 15385 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55fb2775f4d64632a2eed20429002df7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fb2775f4d64632a2eed20429002df7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34141 } }
I20260810 23:30:41.187225 15385 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55fb2775f4d64632a2eed20429002df7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:30:41.187357 15385 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55fb2775f4d64632a2eed20429002df7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:30:41.187534 15385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55fb2775f4d64632a2eed20429002df7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:30:41.191241 15385 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55fb2775f4d64632a2eed20429002df7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fb2775f4d64632a2eed20429002df7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34141 } }
I20260810 23:30:41.192034 15385 leader_election.cc:302] T 00000000000000000000000000000000 P 55fb2775f4d64632a2eed20429002df7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55fb2775f4d64632a2eed20429002df7; no voters:
I20260810 23:30:41.192925 15385 leader_election.cc:288] T 00000000000000000000000000000000 P 55fb2775f4d64632a2eed20429002df7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:30:41.193176 15396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55fb2775f4d64632a2eed20429002df7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:30:41.193879 15396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55fb2775f4d64632a2eed20429002df7 [term 1 LEADER]: Becoming Leader. State: Replica: 55fb2775f4d64632a2eed20429002df7, State: Running, Role: LEADER
I20260810 23:30:41.194531 15396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55fb2775f4d64632a2eed20429002df7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fb2775f4d64632a2eed20429002df7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34141 } }
I20260810 23:30:41.196484 15396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55fb2775f4d64632a2eed20429002df7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55fb2775f4d64632a2eed20429002df7. Latest consensus state: current_term: 1 leader_uuid: "55fb2775f4d64632a2eed20429002df7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fb2775f4d64632a2eed20429002df7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34141 } } }
I20260810 23:30:41.196789 15396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55fb2775f4d64632a2eed20429002df7 [sys.catalog]: This master's current role is: LEADER
I20260810 23:30:41.197307 15398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55fb2775f4d64632a2eed20429002df7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55fb2775f4d64632a2eed20429002df7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fb2775f4d64632a2eed20429002df7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34141 } } }
I20260810 23:30:41.197683 15398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55fb2775f4d64632a2eed20429002df7 [sys.catalog]: This master's current role is: LEADER
I20260810 23:30:41.202102 15385 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55fb2775f4d64632a2eed20429002df7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:30:41.206903 15401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:30:41.213201 15401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:30:41.228425 15389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:41.206282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60807 (local address 127.0.0.1:34141)
0810 23:30:41.206483 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:41.206490 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:41.207647 (+ 1157us) server_negotiation.cc:407] Connection header received
0810 23:30:41.207978 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:41.207985 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:41.208075 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:41.208158 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:41.210387 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:41.211599 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:41.213604 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:41.214335 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:41.214457 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:41.219241 (+ 4784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:41.219249 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:41.219254 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:30:41.219304 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:41.223502 (+ 4198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:41.224967 (+ 1465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:41.224975 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:41.224980 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:41.225218 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:41.225978 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:41.225984 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:41.225988 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:41.226496 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:41.226820 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:41.227698 (+ 878us) server_negotiation.cc:299] Negotiation successful
0810 23:30:41.227959 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 23:30:41.235512 15401 catalog_manager.cc:1347] Generated new cluster ID: 7cd0326e29b14ed78be267b6355f5737
I20260810 23:30:41.235673 15401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:30:41.261328 15401 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:30:41.262701 15401 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:30:41.271288 15401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55fb2775f4d64632a2eed20429002df7: Generated new TSK 0
I20260810 23:30:41.272193 15401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:30:41.278858 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 15275
I20260810 23:30:41.280270 15378 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:30:41.374207 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 15275
W20260810 23:30:41.400123 15255 connection.cc:535] client connection to 127.0.0.1:34141 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34141): Connection reset by peer (error 104)
2026-08-10T23:30:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2344 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:30:41 dist-test-slave-rmhg krb5kdc[15429](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:30:41 dist-test-slave-rmhg krb5kdc[15429](info): set up 2 sockets
Aug 10 23:30:41 dist-test-slave-rmhg krb5kdc[15429](info): commencing operation
krb5kdc: starting...
W20260810 23:30:42.628671 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.209s user 0.000s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:30:42 dist-test-slave-rmhg krb5kdc[15429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:30:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:30:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:30:42.958722 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35135
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:30:43.316217 15446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:30:43.316540 15446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:30:43.316666 15446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:30:43.324240 15446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:30:43.324378 15446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:30:43.324455 15446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:30:43.324577 15446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:30:43.324637 15446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:30:43.335260 15446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35135
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:30:43.336788 15446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:30:43.338382 15446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:30:43.351758 15454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:43.351807 15453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:43.352481 15457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:30:43.352456 15446 server_base.cc:1034] running on GCE node
I20260810 23:30:43.353474 15446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:30:43.355394 15446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:30:43.356607 15446 hybrid_clock.cc:648] HybridClock initialized: now 1786404643356569 us; error 32 us; skew 500 ppm
I20260810 23:30:43.356984 15446 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:30:43 dist-test-slave-rmhg krb5kdc[15429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:30:43.360252 15446 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:30:43.361935 15446 webserver.cc:460] Webserver started at http://127.0.0.1:39701/ using document root <none> and password file <none>
I20260810 23:30:43.362643 15446 fs_manager.cc:359] Metadata directory not provided
I20260810 23:30:43.362766 15446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:30:43.363065 15446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:30:43.366302 15446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "02eeb23e1a9e46beb84f6523192cb1d6"
format_stamp: "Formatted at 2026-08-10 23:30:43 on dist-test-slave-rmhg"
server_key: "81fba1e6e862287b7629cef9d9d1059e"
server_key_iv: "3cec965cde017553d5488882792a65eb"
server_key_version: "encryptionkey@0"
I20260810 23:30:43.367132 15446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "02eeb23e1a9e46beb84f6523192cb1d6"
format_stamp: "Formatted at 2026-08-10 23:30:43 on dist-test-slave-rmhg"
server_key: "81fba1e6e862287b7629cef9d9d1059e"
server_key_iv: "3cec965cde017553d5488882792a65eb"
server_key_version: "encryptionkey@0"
I20260810 23:30:43.373143 15446 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260810 23:30:43.377288 15469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:43.378844 15446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:30:43.379040 15446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "02eeb23e1a9e46beb84f6523192cb1d6"
format_stamp: "Formatted at 2026-08-10 23:30:43 on dist-test-slave-rmhg"
server_key: "81fba1e6e862287b7629cef9d9d1059e"
server_key_iv: "3cec965cde017553d5488882792a65eb"
server_key_version: "encryptionkey@0"
I20260810 23:30:43.379257 15446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:30:43.397375 15446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:30:43.401904 15446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:30:43.402215 15446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:30:43.421384 15446 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35135
I20260810 23:30:43.421403 15544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35135 every 8 connection(s)
I20260810 23:30:43.422879 15446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:30:43.426826 15546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:30:43.432513 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 15446
I20260810 23:30:43.432720 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:30:43.433216 15037 external_mini_cluster.cc:1428] Setting key abd18bccc24802515c03e4d3f3fb2fb4
I20260810 23:30:43.438220 15546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02eeb23e1a9e46beb84f6523192cb1d6: Bootstrap starting.
I20260810 23:30:43.442008 15546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02eeb23e1a9e46beb84f6523192cb1d6: Neither blocks nor log segments found. Creating new log.
I20260810 23:30:43.443411 15546 log.cc:824] T 00000000000000000000000000000000 P 02eeb23e1a9e46beb84f6523192cb1d6: Log is configured to *not* fsync() on all Append() calls
I20260810 23:30:43.446817 15546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02eeb23e1a9e46beb84f6523192cb1d6: No bootstrap required, opened a new log
I20260810 23:30:43.447583 15549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:43.435190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54931 (local address 127.0.0.1:35135)
0810 23:30:43.436068 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:43.436083 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:43.436142 (+ 59us) server_negotiation.cc:407] Connection header received
0810 23:30:43.437947 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:43.437973 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:43.438435 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:43.438791 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:43.439395 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:43.440761 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:43.442685 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:43.443808 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:43.443957 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:43.445259 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:43.445267 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:43.445283 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:43.445838 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:43.445915 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:43.445941 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:43.446058 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:43.446328 (+ 270us) server_negotiation.cc:299] Negotiation successful
0810 23:30:43.446501 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":407,"threads_started":1}
I20260810 23:30:43.451507 15037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:30:43.456573 15546 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02eeb23e1a9e46beb84f6523192cb1d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02eeb23e1a9e46beb84f6523192cb1d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35135 } }
I20260810 23:30:43.457070 15546 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02eeb23e1a9e46beb84f6523192cb1d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:30:43.457302 15546 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02eeb23e1a9e46beb84f6523192cb1d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02eeb23e1a9e46beb84f6523192cb1d6, State: Initialized, Role: FOLLOWER
I20260810 23:30:43.458086 15546 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02eeb23e1a9e46beb84f6523192cb1d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02eeb23e1a9e46beb84f6523192cb1d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35135 } }
I20260810 23:30:43.458395 15546 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02eeb23e1a9e46beb84f6523192cb1d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:30:43.458555 15546 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02eeb23e1a9e46beb84f6523192cb1d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:30:43.458770 15546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02eeb23e1a9e46beb84f6523192cb1d6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:30:43.461875 15546 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02eeb23e1a9e46beb84f6523192cb1d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02eeb23e1a9e46beb84f6523192cb1d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35135 } }
I20260810 23:30:43.462502 15546 leader_election.cc:302] T 00000000000000000000000000000000 P 02eeb23e1a9e46beb84f6523192cb1d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02eeb23e1a9e46beb84f6523192cb1d6; no voters:
I20260810 23:30:43.463195 15546 leader_election.cc:288] T 00000000000000000000000000000000 P 02eeb23e1a9e46beb84f6523192cb1d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:30:43.463287 15560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02eeb23e1a9e46beb84f6523192cb1d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:30:43.464560 15549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:43.457112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49769 (local address 127.0.0.1:35135)
0810 23:30:43.457347 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:43.457355 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:43.457830 (+ 475us) server_negotiation.cc:407] Connection header received
0810 23:30:43.458042 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:43.458045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:43.458114 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:43.458192 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:43.458905 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:43.459710 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:43.461617 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:43.462105 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:43.462197 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:43.463475 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:43.463482 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:43.463486 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:43.463514 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:43.463598 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:43.463636 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:43.463769 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:43.464090 (+ 321us) server_negotiation.cc:299] Negotiation successful
0810 23:30:43.464160 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 23:30:43.464228 15560 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02eeb23e1a9e46beb84f6523192cb1d6 [term 1 LEADER]: Becoming Leader. State: Replica: 02eeb23e1a9e46beb84f6523192cb1d6, State: Running, Role: LEADER
I20260810 23:30:43.465345 15560 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02eeb23e1a9e46beb84f6523192cb1d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02eeb23e1a9e46beb84f6523192cb1d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35135 } }
I20260810 23:30:43.465770 15546 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02eeb23e1a9e46beb84f6523192cb1d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:30:43.468358 15561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02eeb23e1a9e46beb84f6523192cb1d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02eeb23e1a9e46beb84f6523192cb1d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02eeb23e1a9e46beb84f6523192cb1d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35135 } } }
I20260810 23:30:43.468442 15563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02eeb23e1a9e46beb84f6523192cb1d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02eeb23e1a9e46beb84f6523192cb1d6. Latest consensus state: current_term: 1 leader_uuid: "02eeb23e1a9e46beb84f6523192cb1d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02eeb23e1a9e46beb84f6523192cb1d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35135 } } }
I20260810 23:30:43.468722 15561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02eeb23e1a9e46beb84f6523192cb1d6 [sys.catalog]: This master's current role is: LEADER
I20260810 23:30:43.468796 15563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02eeb23e1a9e46beb84f6523192cb1d6 [sys.catalog]: This master's current role is: LEADER
I20260810 23:30:43.470106 15573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:30:43.475847 15573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:30:43.486757 15573 catalog_manager.cc:1347] Generated new cluster ID: 9f14a9ebfd064fccb2254be0d07ea34d
I20260810 23:30:43.486912 15573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:30:43.526902 15573 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:30:43.528481 15573 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:30:43.538787 15573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02eeb23e1a9e46beb84f6523192cb1d6: Generated new TSK 0
I20260810 23:30:43.539733 15573 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:30:43.547636 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 15446
W20260810 23:30:43.548955 15539 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54931
W20260810 23:30:43.549854 15037 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 15446: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:30:43.550060 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 15446
W20260810 23:30:43.574867 15426 connection.cc:535] client connection to 127.0.0.1:35135 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35135): Connection reset by peer (error 104)
2026-08-10T23:30:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2174 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:30:43 dist-test-slave-rmhg krb5kdc[15590](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:30:43 dist-test-slave-rmhg krb5kdc[15590](info): set up 2 sockets
Aug 10 23:30:43 dist-test-slave-rmhg krb5kdc[15590](info): commencing operation
krb5kdc: starting...
W20260810 23:30:44.799292 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.205s user 0.004s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:30:45 dist-test-slave-rmhg krb5kdc[15590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:30:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:30:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:30:45.146380 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43449
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:30:45.506278 15615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:30:45.506608 15615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:30:45.506744 15615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:30:45.514513 15615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:30:45.514665 15615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:30:45.514721 15615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:30:45.514809 15615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:30:45.514864 15615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:30:45.525521 15615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43449
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:30:45.526978 15615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:30:45.528666 15615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:30:45.541765 15621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:45.541706 15622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:45.541738 15624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:30:45.542804 15615 server_base.cc:1034] running on GCE node
I20260810 23:30:45.543669 15615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:30:45.545288 15615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:30:45.546542 15615 hybrid_clock.cc:648] HybridClock initialized: now 1786404645546482 us; error 46 us; skew 500 ppm
I20260810 23:30:45.546937 15615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:30:45 dist-test-slave-rmhg krb5kdc[15590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:30:45.550421 15615 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:30:45.552037 15615 webserver.cc:460] Webserver started at http://127.0.0.1:42289/ using document root <none> and password file <none>
I20260810 23:30:45.552737 15615 fs_manager.cc:359] Metadata directory not provided
I20260810 23:30:45.552847 15615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:30:45.553145 15615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:30:45.556471 15615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "0b94db746e3e49c3b26cf76d257bb628"
format_stamp: "Formatted at 2026-08-10 23:30:45 on dist-test-slave-rmhg"
server_key: "787e20286f2855c8087560a64eeafd74"
server_key_iv: "fbdf208d8051d58565f798a52ee02a26"
server_key_version: "encryptionkey@0"
I20260810 23:30:45.557332 15615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "0b94db746e3e49c3b26cf76d257bb628"
format_stamp: "Formatted at 2026-08-10 23:30:45 on dist-test-slave-rmhg"
server_key: "787e20286f2855c8087560a64eeafd74"
server_key_iv: "fbdf208d8051d58565f798a52ee02a26"
server_key_version: "encryptionkey@0"
I20260810 23:30:45.562721 15615 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260810 23:30:45.566300 15633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:45.567664 15615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:30:45.567836 15615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "0b94db746e3e49c3b26cf76d257bb628"
format_stamp: "Formatted at 2026-08-10 23:30:45 on dist-test-slave-rmhg"
server_key: "787e20286f2855c8087560a64eeafd74"
server_key_iv: "fbdf208d8051d58565f798a52ee02a26"
server_key_version: "encryptionkey@0"
I20260810 23:30:45.568041 15615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:30:45.574157 15615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:30:45.578495 15615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:30:45.578804 15615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:30:45.598433 15615 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43449
I20260810 23:30:45.598483 15710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43449 every 8 connection(s)
I20260810 23:30:45.599984 15615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:30:45.604038 15712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:30:45.609731 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 15615
I20260810 23:30:45.609906 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:30:45.610359 15037 external_mini_cluster.cc:1428] Setting key 52540a0245027fe2225f4a8c64c0d75e
I20260810 23:30:45.615744 15712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b94db746e3e49c3b26cf76d257bb628: Bootstrap starting.
I20260810 23:30:45.618860 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:45.612324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40333 (local address 127.0.0.1:43449)
0810 23:30:45.613340 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:45.613354 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:45.613406 (+ 52us) server_negotiation.cc:407] Connection header received
0810 23:30:45.615189 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:45.615215 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:45.615603 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:45.616004 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:45.617098 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:45.617105 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:45.617119 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:45.617135 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:45.617245 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:45.617278 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:45.617516 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:45.617658 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 23:30:45.617877 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":432,"threads_started":1}
I20260810 23:30:45.619263 15712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b94db746e3e49c3b26cf76d257bb628: Neither blocks nor log segments found. Creating new log.
I20260810 23:30:45.620409 15712 log.cc:824] T 00000000000000000000000000000000 P 0b94db746e3e49c3b26cf76d257bb628: Log is configured to *not* fsync() on all Append() calls
I20260810 23:30:45.622888 15037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:30:45.623769 15712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b94db746e3e49c3b26cf76d257bb628: No bootstrap required, opened a new log
I20260810 23:30:45.630148 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:45.627364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55783 (local address 127.0.0.1:43449)
0810 23:30:45.627562 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:45.627570 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:45.627923 (+ 353us) server_negotiation.cc:407] Connection header received
0810 23:30:45.628083 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:45.628089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:45.628153 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:45.628276 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:45.629326 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:45.629331 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:45.629336 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:45.629357 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:45.629410 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:45.629435 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:45.629604 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:45.629736 (+ 132us) server_negotiation.cc:299] Negotiation successful
0810 23:30:45.629802 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:30:45.632638 15712 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b94db746e3e49c3b26cf76d257bb628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b94db746e3e49c3b26cf76d257bb628" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43449 } }
I20260810 23:30:45.633060 15712 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b94db746e3e49c3b26cf76d257bb628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:30:45.633172 15712 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b94db746e3e49c3b26cf76d257bb628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b94db746e3e49c3b26cf76d257bb628, State: Initialized, Role: FOLLOWER
I20260810 23:30:45.633826 15712 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b94db746e3e49c3b26cf76d257bb628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b94db746e3e49c3b26cf76d257bb628" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43449 } }
I20260810 23:30:45.634063 15712 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b94db746e3e49c3b26cf76d257bb628 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:30:45.634174 15712 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b94db746e3e49c3b26cf76d257bb628 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:30:45.634320 15712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b94db746e3e49c3b26cf76d257bb628 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:30:45.637228 15712 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b94db746e3e49c3b26cf76d257bb628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b94db746e3e49c3b26cf76d257bb628" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43449 } }
I20260810 23:30:45.637781 15712 leader_election.cc:302] T 00000000000000000000000000000000 P 0b94db746e3e49c3b26cf76d257bb628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b94db746e3e49c3b26cf76d257bb628; no voters:
I20260810 23:30:45.638542 15712 leader_election.cc:288] T 00000000000000000000000000000000 P 0b94db746e3e49c3b26cf76d257bb628 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:30:45.638729 15726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b94db746e3e49c3b26cf76d257bb628 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:30:45.639539 15726 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b94db746e3e49c3b26cf76d257bb628 [term 1 LEADER]: Becoming Leader. State: Replica: 0b94db746e3e49c3b26cf76d257bb628, State: Running, Role: LEADER
I20260810 23:30:45.640213 15726 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b94db746e3e49c3b26cf76d257bb628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b94db746e3e49c3b26cf76d257bb628" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43449 } }
I20260810 23:30:45.641279 15712 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b94db746e3e49c3b26cf76d257bb628 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:30:45.643229 15728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b94db746e3e49c3b26cf76d257bb628 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b94db746e3e49c3b26cf76d257bb628. Latest consensus state: current_term: 1 leader_uuid: "0b94db746e3e49c3b26cf76d257bb628" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b94db746e3e49c3b26cf76d257bb628" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43449 } } }
I20260810 23:30:45.643734 15728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b94db746e3e49c3b26cf76d257bb628 [sys.catalog]: This master's current role is: LEADER
I20260810 23:30:45.643363 15730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b94db746e3e49c3b26cf76d257bb628 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b94db746e3e49c3b26cf76d257bb628" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b94db746e3e49c3b26cf76d257bb628" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43449 } } }
I20260810 23:30:45.644097 15730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b94db746e3e49c3b26cf76d257bb628 [sys.catalog]: This master's current role is: LEADER
I20260810 23:30:45.654789 15739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:30:45.659819 15739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:30:45.669396 15739 catalog_manager.cc:1347] Generated new cluster ID: 150f0a0530c44fa0935e63fb42a2aa03
I20260810 23:30:45.669549 15739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:30:45.684662 15739 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:30:45.685330 15739 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:30:45.696736 15739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b94db746e3e49c3b26cf76d257bb628: Generated new TSK 0
I20260810 23:30:45.697962 15739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:30:45.710999 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 15615
W20260810 23:30:45.712494 15703 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40333
W20260810 23:30:45.713374 15037 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 15615: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:30:45.713537 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 15615
2026-08-10T23:30:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2168 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:30:45 dist-test-slave-rmhg krb5kdc[15760](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:30:45 dist-test-slave-rmhg krb5kdc[15760](info): set up 2 sockets
Aug 10 23:30:45 dist-test-slave-rmhg krb5kdc[15760](info): commencing operation
krb5kdc: starting...
W20260810 23:30:46.974599 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.212s user 0.000s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:30:47 dist-test-slave-rmhg krb5kdc[15760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:30:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:30:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:30:47.304054 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38177
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:30:47.654081 15779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:30:47.654392 15779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:30:47.654515 15779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:30:47.662001 15779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:30:47.662145 15779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:30:47.662242 15779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:30:47.662329 15779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:30:47.662387 15779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:30:47.672830 15779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:30:47.674249 15779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:30:47.675617 15779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:30:47.689252 15795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:47.689345 15791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:47.689366 15793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:30:47.690465 15779 server_base.cc:1034] running on GCE node
I20260810 23:30:47.691331 15779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:30:47.692879 15779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:30:47.694135 15779 hybrid_clock.cc:648] HybridClock initialized: now 1786404647694064 us; error 55 us; skew 500 ppm
I20260810 23:30:47.694592 15779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:30:47 dist-test-slave-rmhg krb5kdc[15760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:30:47.704926 15779 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:30:47.706791 15779 webserver.cc:460] Webserver started at http://127.0.0.1:32779/ using document root <none> and password file <none>
I20260810 23:30:47.707541 15779 fs_manager.cc:359] Metadata directory not provided
I20260810 23:30:47.707695 15779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:30:47.708040 15779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:30:47.711274 15779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "09da269e2e8c43e49dc71b6303514eef"
format_stamp: "Formatted at 2026-08-10 23:30:47 on dist-test-slave-rmhg"
server_key: "55f8bb0a91fadda5185876a5374851f1"
server_key_iv: "3b5b271032daadda31620eaa47e38bf0"
server_key_version: "encryptionkey@0"
I20260810 23:30:47.712090 15779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "09da269e2e8c43e49dc71b6303514eef"
format_stamp: "Formatted at 2026-08-10 23:30:47 on dist-test-slave-rmhg"
server_key: "55f8bb0a91fadda5185876a5374851f1"
server_key_iv: "3b5b271032daadda31620eaa47e38bf0"
server_key_version: "encryptionkey@0"
I20260810 23:30:47.717784 15779 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260810 23:30:47.721472 15804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:47.722909 15779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:30:47.723099 15779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "09da269e2e8c43e49dc71b6303514eef"
format_stamp: "Formatted at 2026-08-10 23:30:47 on dist-test-slave-rmhg"
server_key: "55f8bb0a91fadda5185876a5374851f1"
server_key_iv: "3b5b271032daadda31620eaa47e38bf0"
server_key_version: "encryptionkey@0"
I20260810 23:30:47.723320 15779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:30:47.736824 15779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:30:47.741498 15779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:30:47.741806 15779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:30:47.761433 15779 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38177
I20260810 23:30:47.761519 15878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38177 every 8 connection(s)
I20260810 23:30:47.762970 15779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:30:47.766997 15879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:30:47.767094 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 15779
I20260810 23:30:47.767280 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:30:47.767768 15037 external_mini_cluster.cc:1428] Setting key 7fd29120bbd0f78f32725c8f1d627bdb
I20260810 23:30:47.781273 15879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09da269e2e8c43e49dc71b6303514eef: Bootstrap starting.
Aug 10 23:30:47 dist-test-slave-rmhg krb5kdc[15760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 23:30:47.785507 15879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09da269e2e8c43e49dc71b6303514eef: Neither blocks nor log segments found. Creating new log.
I20260810 23:30:47.786726 15879 log.cc:824] T 00000000000000000000000000000000 P 09da269e2e8c43e49dc71b6303514eef: Log is configured to *not* fsync() on all Append() calls
I20260810 23:30:47.790310 15879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09da269e2e8c43e49dc71b6303514eef: No bootstrap required, opened a new log
I20260810 23:30:47.794214 15885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:47.769629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57327 (local address 127.0.0.1:38177)
0810 23:30:47.770517 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:47.770531 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:47.770575 (+ 44us) server_negotiation.cc:407] Connection header received
0810 23:30:47.773034 (+ 2459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:47.773066 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:47.773585 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:47.773949 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:47.775835 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:47.777247 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:47.779149 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:47.780012 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:47.780169 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:47.785394 (+ 5225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:47.785403 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:47.785421 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:30:47.785466 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:47.789128 (+ 3662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:47.789969 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:47.789978 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:47.789987 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:47.790224 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:47.790711 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:47.790716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:47.790719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:47.791507 (+ 788us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:47.791816 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:47.792207 (+ 391us) server_negotiation.cc:299] Negotiation successful
0810 23:30:47.792711 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":420,"threads_started":1}
I20260810 23:30:47.798252 15037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:30:47.799959 15879 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09da269e2e8c43e49dc71b6303514eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09da269e2e8c43e49dc71b6303514eef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38177 } }
I20260810 23:30:47.800436 15879 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09da269e2e8c43e49dc71b6303514eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:30:47.800539 15879 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09da269e2e8c43e49dc71b6303514eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09da269e2e8c43e49dc71b6303514eef, State: Initialized, Role: FOLLOWER
I20260810 23:30:47.801276 15879 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09da269e2e8c43e49dc71b6303514eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09da269e2e8c43e49dc71b6303514eef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38177 } }
I20260810 23:30:47.801576 15879 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09da269e2e8c43e49dc71b6303514eef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:30:47.801728 15879 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09da269e2e8c43e49dc71b6303514eef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:30:47.801911 15879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09da269e2e8c43e49dc71b6303514eef [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:30:47.804912 15879 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09da269e2e8c43e49dc71b6303514eef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09da269e2e8c43e49dc71b6303514eef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38177 } }
I20260810 23:30:47.805447 15879 leader_election.cc:302] T 00000000000000000000000000000000 P 09da269e2e8c43e49dc71b6303514eef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09da269e2e8c43e49dc71b6303514eef; no voters:
I20260810 23:30:47.806082 15879 leader_election.cc:288] T 00000000000000000000000000000000 P 09da269e2e8c43e49dc71b6303514eef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:30:47.806195 15897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09da269e2e8c43e49dc71b6303514eef [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:30:47.807366 15897 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09da269e2e8c43e49dc71b6303514eef [term 1 LEADER]: Becoming Leader. State: Replica: 09da269e2e8c43e49dc71b6303514eef, State: Running, Role: LEADER
I20260810 23:30:47.808053 15897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09da269e2e8c43e49dc71b6303514eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09da269e2e8c43e49dc71b6303514eef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38177 } }
I20260810 23:30:47.808616 15879 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09da269e2e8c43e49dc71b6303514eef [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:30:47.813349 15901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09da269e2e8c43e49dc71b6303514eef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09da269e2e8c43e49dc71b6303514eef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09da269e2e8c43e49dc71b6303514eef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38177 } } }
I20260810 23:30:47.813602 15904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09da269e2e8c43e49dc71b6303514eef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09da269e2e8c43e49dc71b6303514eef. Latest consensus state: current_term: 1 leader_uuid: "09da269e2e8c43e49dc71b6303514eef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09da269e2e8c43e49dc71b6303514eef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38177 } } }
I20260810 23:30:47.813800 15901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09da269e2e8c43e49dc71b6303514eef [sys.catalog]: This master's current role is: LEADER
I20260810 23:30:47.813874 15904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09da269e2e8c43e49dc71b6303514eef [sys.catalog]: This master's current role is: LEADER
I20260810 23:30:47.821318 15885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:47.803705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52679 (local address 127.0.0.1:38177)
0810 23:30:47.803908 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:47.803914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:47.804313 (+ 399us) server_negotiation.cc:407] Connection header received
0810 23:30:47.804455 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:47.804473 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:47.804560 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:47.804672 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:47.806320 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:47.807468 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:47.808840 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:47.809382 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:47.809512 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:47.812262 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:47.812272 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:47.812278 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:30:47.812333 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:47.816518 (+ 4185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:47.817665 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:47.817671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:47.817676 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:47.817865 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:47.818377 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:47.818382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:47.818386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:47.818888 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:47.819169 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:47.820595 (+ 1426us) server_negotiation.cc:299] Negotiation successful
0810 23:30:47.820870 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 23:30:47.828913 15912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:30:47.834687 15912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:30:47.844118 15912 catalog_manager.cc:1347] Generated new cluster ID: 6064752bb7dc4d5bbe8df4d4d31b7552
I20260810 23:30:47.844259 15912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:30:47.863359 15912 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:30:47.864856 15912 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:30:47.872426 15912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09da269e2e8c43e49dc71b6303514eef: Generated new TSK 0
I20260810 23:30:47.873337 15912 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:30:47.881939 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 15779
I20260810 23:30:47.883018 15872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:30:47.975642 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 15779
2026-08-10T23:30:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2259 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:30:48 dist-test-slave-rmhg krb5kdc[15938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:30:48 dist-test-slave-rmhg krb5kdc[15938](info): set up 2 sockets
Aug 10 23:30:48 dist-test-slave-rmhg krb5kdc[15938](info): commencing operation
krb5kdc: starting...
W20260810 23:30:49.236553 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.214s user 0.005s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:30:49 dist-test-slave-rmhg krb5kdc[15938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:30:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:30:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:30:49.568532 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46851
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:30:49.926626 15955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:30:49.926970 15955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:30:49.927090 15955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:30:49.934839 15955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:30:49.934983 15955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:30:49.935037 15955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:30:49.935101 15955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:30:49.935148 15955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:30:49.945734 15955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:30:49.947089 15955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:30:49.948444 15955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:30:49.961028 15972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:49.961093 15966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:49.961124 15970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:30:49.962057 15955 server_base.cc:1034] running on GCE node
I20260810 23:30:49.962942 15955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:30:49.964555 15955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:30:49.965770 15955 hybrid_clock.cc:648] HybridClock initialized: now 1786404649965719 us; error 40 us; skew 500 ppm
I20260810 23:30:49.966146 15955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:30:49 dist-test-slave-rmhg krb5kdc[15938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:30:49.969532 15955 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:30:49.971117 15955 webserver.cc:460] Webserver started at http://127.0.0.1:40557/ using document root <none> and password file <none>
I20260810 23:30:49.971822 15955 fs_manager.cc:359] Metadata directory not provided
I20260810 23:30:49.971947 15955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:30:49.972250 15955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:30:49.975449 15955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "84bf20924fbd418c83c008c9b8ab8ed8"
format_stamp: "Formatted at 2026-08-10 23:30:49 on dist-test-slave-rmhg"
server_key: "dca8f552f8044050042c6b5649cd5625"
server_key_iv: "8095955abcbfad8b1fd8c911d2b3a471"
server_key_version: "encryptionkey@0"
I20260810 23:30:49.976287 15955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "84bf20924fbd418c83c008c9b8ab8ed8"
format_stamp: "Formatted at 2026-08-10 23:30:49 on dist-test-slave-rmhg"
server_key: "dca8f552f8044050042c6b5649cd5625"
server_key_iv: "8095955abcbfad8b1fd8c911d2b3a471"
server_key_version: "encryptionkey@0"
I20260810 23:30:49.982009 15955 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 23:30:49.987733 15980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:49.989364 15955 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.006s
I20260810 23:30:49.989557 15955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "84bf20924fbd418c83c008c9b8ab8ed8"
format_stamp: "Formatted at 2026-08-10 23:30:49 on dist-test-slave-rmhg"
server_key: "dca8f552f8044050042c6b5649cd5625"
server_key_iv: "8095955abcbfad8b1fd8c911d2b3a471"
server_key_version: "encryptionkey@0"
I20260810 23:30:49.989773 15955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:30:50.016021 15955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:30:50.020561 15955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:30:50.020856 15955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:30:50.039739 15955 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46851
I20260810 23:30:50.039762 16058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46851 every 8 connection(s)
I20260810 23:30:50.041249 15955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:30:50.041996 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 15955
I20260810 23:30:50.042168 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:30:50.042732 15037 external_mini_cluster.cc:1428] Setting key f682df78d22e6a7a2e06417c63e77c0f
I20260810 23:30:50.046275 16059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:30:50.059037 16062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:50.044683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49517 (local address 127.0.0.1:46851)
0810 23:30:50.045617 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:50.045633 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:50.045689 (+ 56us) server_negotiation.cc:407] Connection header received
0810 23:30:50.048112 (+ 2423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:50.048148 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:50.048748 (+ 600us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:50.049273 (+ 525us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:50.049876 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:50.051944 (+ 2068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:50.053426 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:50.054499 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:50.054685 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:50.055832 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:50.055844 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:50.055865 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:50.056633 (+ 768us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:50.056777 (+ 144us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:50.056818 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:50.056938 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:50.057244 (+ 306us) server_negotiation.cc:299] Negotiation successful
0810 23:30:50.057586 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":412,"threads_started":1}
I20260810 23:30:50.061486 16059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84bf20924fbd418c83c008c9b8ab8ed8: Bootstrap starting.
I20260810 23:30:50.063429 15037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:30:50.065529 16059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84bf20924fbd418c83c008c9b8ab8ed8: Neither blocks nor log segments found. Creating new log.
I20260810 23:30:50.066933 16059 log.cc:824] T 00000000000000000000000000000000 P 84bf20924fbd418c83c008c9b8ab8ed8: Log is configured to *not* fsync() on all Append() calls
I20260810 23:30:50.070559 16059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84bf20924fbd418c83c008c9b8ab8ed8: No bootstrap required, opened a new log
I20260810 23:30:50.075810 16062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:50.068743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49973 (local address 127.0.0.1:46851)
0810 23:30:50.068916 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:50.068923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:50.069456 (+ 533us) server_negotiation.cc:407] Connection header received
0810 23:30:50.069691 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:50.069695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:50.069758 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:50.069837 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:50.070477 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:50.071272 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:50.073209 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:50.073696 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:50.073804 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:50.074979 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:50.074984 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:50.074987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:50.075007 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:50.075047 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:50.075063 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:50.075133 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:50.075273 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 23:30:50.075355 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 23:30:50.080173 16059 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84bf20924fbd418c83c008c9b8ab8ed8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84bf20924fbd418c83c008c9b8ab8ed8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46851 } }
I20260810 23:30:50.080603 16059 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84bf20924fbd418c83c008c9b8ab8ed8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:30:50.080716 16059 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84bf20924fbd418c83c008c9b8ab8ed8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84bf20924fbd418c83c008c9b8ab8ed8, State: Initialized, Role: FOLLOWER
I20260810 23:30:50.081377 16059 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84bf20924fbd418c83c008c9b8ab8ed8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84bf20924fbd418c83c008c9b8ab8ed8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46851 } }
I20260810 23:30:50.081595 16059 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84bf20924fbd418c83c008c9b8ab8ed8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:30:50.081705 16059 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84bf20924fbd418c83c008c9b8ab8ed8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:30:50.081832 16059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84bf20924fbd418c83c008c9b8ab8ed8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:30:50.084532 16059 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84bf20924fbd418c83c008c9b8ab8ed8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84bf20924fbd418c83c008c9b8ab8ed8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46851 } }
I20260810 23:30:50.085043 16059 leader_election.cc:302] T 00000000000000000000000000000000 P 84bf20924fbd418c83c008c9b8ab8ed8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84bf20924fbd418c83c008c9b8ab8ed8; no voters:
I20260810 23:30:50.085667 16059 leader_election.cc:288] T 00000000000000000000000000000000 P 84bf20924fbd418c83c008c9b8ab8ed8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:30:50.085772 16078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84bf20924fbd418c83c008c9b8ab8ed8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:30:50.086476 16078 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84bf20924fbd418c83c008c9b8ab8ed8 [term 1 LEADER]: Becoming Leader. State: Replica: 84bf20924fbd418c83c008c9b8ab8ed8, State: Running, Role: LEADER
I20260810 23:30:50.087060 16078 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84bf20924fbd418c83c008c9b8ab8ed8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84bf20924fbd418c83c008c9b8ab8ed8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46851 } }
I20260810 23:30:50.087821 16059 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84bf20924fbd418c83c008c9b8ab8ed8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:30:50.089890 16081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84bf20924fbd418c83c008c9b8ab8ed8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84bf20924fbd418c83c008c9b8ab8ed8. Latest consensus state: current_term: 1 leader_uuid: "84bf20924fbd418c83c008c9b8ab8ed8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84bf20924fbd418c83c008c9b8ab8ed8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46851 } } }
I20260810 23:30:50.090317 16081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84bf20924fbd418c83c008c9b8ab8ed8 [sys.catalog]: This master's current role is: LEADER
I20260810 23:30:50.091076 16080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84bf20924fbd418c83c008c9b8ab8ed8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84bf20924fbd418c83c008c9b8ab8ed8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84bf20924fbd418c83c008c9b8ab8ed8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46851 } } }
I20260810 23:30:50.091388 16080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84bf20924fbd418c83c008c9b8ab8ed8 [sys.catalog]: This master's current role is: LEADER
I20260810 23:30:50.094933 16090 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:30:50.101845 16090 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:30:50.112936 16090 catalog_manager.cc:1347] Generated new cluster ID: de603451cc894bbf98440e3a70c67400
I20260810 23:30:50.113081 16090 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:30:50.139871 16090 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:30:50.141435 16090 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:30:50.149902 16090 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84bf20924fbd418c83c008c9b8ab8ed8: Generated new TSK 0
I20260810 23:30:50.150844 16090 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:30:50.160159 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 15955
W20260810 23:30:50.161357 16051 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49517
W20260810 23:30:50.162123 15037 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 15955: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:30:50.162259 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 15955
2026-08-10T23:30:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2187 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:30:50 dist-test-slave-rmhg krb5kdc[16120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:30:50 dist-test-slave-rmhg krb5kdc[16120](info): set up 2 sockets
Aug 10 23:30:50 dist-test-slave-rmhg krb5kdc[16120](info): commencing operation
krb5kdc: starting...
W20260810 23:30:51.434666 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.224s user 0.005s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:30:51 dist-test-slave-rmhg krb5kdc[16120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:30:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:30:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:30:51.769902 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37857
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:30:52.122670 16140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:30:52.123131 16140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:30:52.123263 16140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:30:52.130887 16140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:30:52.131031 16140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:30:52.131078 16140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:30:52.131145 16140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:30:52.131189 16140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:30:52.141604 16140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:30:52.142988 16140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:30:52.144341 16140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:30:52.156435 16148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:52.156502 16147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:52.156499 16150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:30:52.157554 16140 server_base.cc:1034] running on GCE node
I20260810 23:30:52.158401 16140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:30:52.159940 16140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:30:52.161190 16140 hybrid_clock.cc:648] HybridClock initialized: now 1786404652161106 us; error 41 us; skew 500 ppm
I20260810 23:30:52.161583 16140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:30:52 dist-test-slave-rmhg krb5kdc[16120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:30:52.165233 16140 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:30:52.166920 16140 webserver.cc:460] Webserver started at http://127.0.0.1:46705/ using document root <none> and password file <none>
I20260810 23:30:52.167654 16140 fs_manager.cc:359] Metadata directory not provided
I20260810 23:30:52.167768 16140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:30:52.168082 16140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:30:52.171640 16140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "4c99ceddb25e4f44ab3275054b1302cd"
format_stamp: "Formatted at 2026-08-10 23:30:52 on dist-test-slave-rmhg"
server_key: "ead8fdde3af5a2a2088e56f526f6da48"
server_key_iv: "529091a304260d7cf58235b361580e38"
server_key_version: "encryptionkey@0"
I20260810 23:30:52.172458 16140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "4c99ceddb25e4f44ab3275054b1302cd"
format_stamp: "Formatted at 2026-08-10 23:30:52 on dist-test-slave-rmhg"
server_key: "ead8fdde3af5a2a2088e56f526f6da48"
server_key_iv: "529091a304260d7cf58235b361580e38"
server_key_version: "encryptionkey@0"
I20260810 23:30:52.178089 16140 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260810 23:30:52.181686 16163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:52.183117 16140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 23:30:52.183302 16140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "4c99ceddb25e4f44ab3275054b1302cd"
format_stamp: "Formatted at 2026-08-10 23:30:52 on dist-test-slave-rmhg"
server_key: "ead8fdde3af5a2a2088e56f526f6da48"
server_key_iv: "529091a304260d7cf58235b361580e38"
server_key_version: "encryptionkey@0"
I20260810 23:30:52.183503 16140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:30:52.189493 16140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:30:52.193611 16140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:30:52.193889 16140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:30:52.213393 16140 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37857
I20260810 23:30:52.213452 16247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37857 every 8 connection(s)
I20260810 23:30:52.214861 16140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:30:52.219136 16249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:30:52.224442 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 16140
I20260810 23:30:52.224695 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:30:52.225298 15037 external_mini_cluster.cc:1428] Setting key c0f2d7f410df888822a47cdf0cdcf062
I20260810 23:30:52.232802 16249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c99ceddb25e4f44ab3275054b1302cd: Bootstrap starting.
I20260810 23:30:52.234860 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:52.227339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46943 (local address 127.0.0.1:37857)
0810 23:30:52.228283 (+ 944us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:52.228301 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:52.228350 (+ 49us) server_negotiation.cc:407] Connection header received
0810 23:30:52.230846 (+ 2496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:52.230885 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:52.231429 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:52.231924 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:52.233187 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:52.233197 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:52.233218 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:52.233238 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:52.233308 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:52.233334 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:52.233548 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:52.233700 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 23:30:52.233896 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"thread_start_us":406,"threads_started":1}
I20260810 23:30:52.236615 16249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c99ceddb25e4f44ab3275054b1302cd: Neither blocks nor log segments found. Creating new log.
I20260810 23:30:52.237908 16249 log.cc:824] T 00000000000000000000000000000000 P 4c99ceddb25e4f44ab3275054b1302cd: Log is configured to *not* fsync() on all Append() calls
I20260810 23:30:52.239202 15037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:30:52.240980 16249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c99ceddb25e4f44ab3275054b1302cd: No bootstrap required, opened a new log
I20260810 23:30:52.246258 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:52.243731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55429 (local address 127.0.0.1:37857)
0810 23:30:52.243926 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:52.243931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:52.244264 (+ 333us) server_negotiation.cc:407] Connection header received
0810 23:30:52.244417 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:52.244422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:52.244517 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:52.244638 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:52.245587 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:52.245593 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:52.245598 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:52.245618 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:52.245675 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:52.245699 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:52.245831 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:52.245874 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 23:30:52.245935 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 23:30:52.249799 16249 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c99ceddb25e4f44ab3275054b1302cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c99ceddb25e4f44ab3275054b1302cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37857 } }
I20260810 23:30:52.250192 16249 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c99ceddb25e4f44ab3275054b1302cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:30:52.250275 16249 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c99ceddb25e4f44ab3275054b1302cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c99ceddb25e4f44ab3275054b1302cd, State: Initialized, Role: FOLLOWER
I20260810 23:30:52.250808 16249 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c99ceddb25e4f44ab3275054b1302cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c99ceddb25e4f44ab3275054b1302cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37857 } }
I20260810 23:30:52.251032 16249 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c99ceddb25e4f44ab3275054b1302cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:30:52.251135 16249 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c99ceddb25e4f44ab3275054b1302cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:30:52.251271 16249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c99ceddb25e4f44ab3275054b1302cd [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:30:52.253569 16249 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c99ceddb25e4f44ab3275054b1302cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c99ceddb25e4f44ab3275054b1302cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37857 } }
I20260810 23:30:52.254115 16249 leader_election.cc:302] T 00000000000000000000000000000000 P 4c99ceddb25e4f44ab3275054b1302cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c99ceddb25e4f44ab3275054b1302cd; no voters:
I20260810 23:30:52.254745 16249 leader_election.cc:288] T 00000000000000000000000000000000 P 4c99ceddb25e4f44ab3275054b1302cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:30:52.254865 16263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c99ceddb25e4f44ab3275054b1302cd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:30:52.255501 16263 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c99ceddb25e4f44ab3275054b1302cd [term 1 LEADER]: Becoming Leader. State: Replica: 4c99ceddb25e4f44ab3275054b1302cd, State: Running, Role: LEADER
I20260810 23:30:52.256060 16263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c99ceddb25e4f44ab3275054b1302cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c99ceddb25e4f44ab3275054b1302cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37857 } }
I20260810 23:30:52.256769 16249 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c99ceddb25e4f44ab3275054b1302cd [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:30:52.258895 16264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c99ceddb25e4f44ab3275054b1302cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c99ceddb25e4f44ab3275054b1302cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c99ceddb25e4f44ab3275054b1302cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37857 } } }
I20260810 23:30:52.258976 16265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c99ceddb25e4f44ab3275054b1302cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c99ceddb25e4f44ab3275054b1302cd. Latest consensus state: current_term: 1 leader_uuid: "4c99ceddb25e4f44ab3275054b1302cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c99ceddb25e4f44ab3275054b1302cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37857 } } }
I20260810 23:30:52.259349 16264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c99ceddb25e4f44ab3275054b1302cd [sys.catalog]: This master's current role is: LEADER
I20260810 23:30:52.259349 16265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c99ceddb25e4f44ab3275054b1302cd [sys.catalog]: This master's current role is: LEADER
I20260810 23:30:52.261406 16279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:30:52.266909 16279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:30:52.276830 16279 catalog_manager.cc:1347] Generated new cluster ID: 6da3b1efe96c48ae80391611c53cfd33
I20260810 23:30:52.276968 16279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:30:52.304901 16279 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:30:52.305418 16279 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:30:52.313527 16279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c99ceddb25e4f44ab3275054b1302cd: Generated new TSK 0
I20260810 23:30:52.314440 16279 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:30:52.326188 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 16140
W20260810 23:30:52.327409 16243 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46943
W20260810 23:30:52.328238 15037 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 16140: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:30:52.328388 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 16140
2026-08-10T23:30:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2163 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:30:52 dist-test-slave-rmhg krb5kdc[16296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:30:52 dist-test-slave-rmhg krb5kdc[16296](info): set up 2 sockets
Aug 10 23:30:52 dist-test-slave-rmhg krb5kdc[16296](info): commencing operation
krb5kdc: starting...
W20260810 23:30:53.580931 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.206s user 0.003s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:30:53 dist-test-slave-rmhg krb5kdc[16296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:30:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:30:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:30:53.912384 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44179
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:30:54.264246 16322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:30:54.264585 16322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:30:54.264729 16322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:30:54.272188 16322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:30:54.272347 16322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:30:54.272434 16322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:30:54.272531 16322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:30:54.272598 16322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:30:54.283085 16322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:30:54.284508 16322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:30:54.285933 16322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:30:54.298125 16329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:54.298161 16330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:54.298206 16333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:30:54.299077 16322 server_base.cc:1034] running on GCE node
I20260810 23:30:54.299917 16322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:30:54.301429 16322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:30:54.302688 16322 hybrid_clock.cc:648] HybridClock initialized: now 1786404654302611 us; error 59 us; skew 500 ppm
I20260810 23:30:54.303089 16322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:30:54 dist-test-slave-rmhg krb5kdc[16296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:30:54.310385 16322 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:30:54.312206 16322 webserver.cc:460] Webserver started at http://127.0.0.1:36113/ using document root <none> and password file <none>
I20260810 23:30:54.312918 16322 fs_manager.cc:359] Metadata directory not provided
I20260810 23:30:54.313055 16322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:30:54.313409 16322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:30:54.316566 16322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "c819904cd64a46c1a615785c043c9c1e"
format_stamp: "Formatted at 2026-08-10 23:30:54 on dist-test-slave-rmhg"
server_key: "ef1811fc6f4e9c5dc1b56aa5238f7361"
server_key_iv: "9c524ad86265dcfa06b082fd3302f7fa"
server_key_version: "encryptionkey@0"
I20260810 23:30:54.317414 16322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "c819904cd64a46c1a615785c043c9c1e"
format_stamp: "Formatted at 2026-08-10 23:30:54 on dist-test-slave-rmhg"
server_key: "ef1811fc6f4e9c5dc1b56aa5238f7361"
server_key_iv: "9c524ad86265dcfa06b082fd3302f7fa"
server_key_version: "encryptionkey@0"
I20260810 23:30:54.322698 16322 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260810 23:30:54.326480 16344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:54.328001 16322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:30:54.328194 16322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "c819904cd64a46c1a615785c043c9c1e"
format_stamp: "Formatted at 2026-08-10 23:30:54 on dist-test-slave-rmhg"
server_key: "ef1811fc6f4e9c5dc1b56aa5238f7361"
server_key_iv: "9c524ad86265dcfa06b082fd3302f7fa"
server_key_version: "encryptionkey@0"
I20260810 23:30:54.328387 16322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:30:54.351408 16322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:30:54.355697 16322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:30:54.355994 16322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:30:54.374807 16322 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44179
I20260810 23:30:54.374848 16429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44179 every 8 connection(s)
I20260810 23:30:54.376605 16322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:30:54.381266 16430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:30:54.384661 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 16322
I20260810 23:30:54.384851 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:30:54.385377 15037 external_mini_cluster.cc:1428] Setting key c5323bd64564b677eb9f408f09a5594b
I20260810 23:30:54.396456 16430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c819904cd64a46c1a615785c043c9c1e: Bootstrap starting.
I20260810 23:30:54.400568 16430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c819904cd64a46c1a615785c043c9c1e: Neither blocks nor log segments found. Creating new log.
I20260810 23:30:54.401782 16430 log.cc:824] T 00000000000000000000000000000000 P c819904cd64a46c1a615785c043c9c1e: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:30:54 dist-test-slave-rmhg krb5kdc[16296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 23:30:54.405246 16430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c819904cd64a46c1a615785c043c9c1e: No bootstrap required, opened a new log
I20260810 23:30:54.412065 16433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:54.387706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:42317 (local address 127.0.0.1:44179)
0810 23:30:54.388793 (+ 1087us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:54.388811 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:54.389202 (+ 391us) server_negotiation.cc:407] Connection header received
0810 23:30:54.391650 (+ 2448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:54.391685 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:54.392322 (+ 637us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:54.392845 (+ 523us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:54.394457 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:54.396605 (+ 2148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:54.398057 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:54.399234 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:54.399439 (+ 205us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:54.404344 (+ 4905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:54.404353 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:54.404370 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:30:54.404408 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:54.407609 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:54.408618 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:54.408626 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:54.408637 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:54.408821 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:54.409366 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:54.409369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:54.409371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:54.409931 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:54.410217 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:54.410684 (+ 467us) server_negotiation.cc:299] Negotiation successful
0810 23:30:54.410952 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":445,"threads_started":1}
I20260810 23:30:54.414194 16430 raft_consensus.cc:348] T 00000000000000000000000000000000 P c819904cd64a46c1a615785c043c9c1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c819904cd64a46c1a615785c043c9c1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44179 } }
I20260810 23:30:54.414582 16430 raft_consensus.cc:374] T 00000000000000000000000000000000 P c819904cd64a46c1a615785c043c9c1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:30:54.414695 16430 raft_consensus.cc:729] T 00000000000000000000000000000000 P c819904cd64a46c1a615785c043c9c1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c819904cd64a46c1a615785c043c9c1e, State: Initialized, Role: FOLLOWER
I20260810 23:30:54.415336 16430 consensus_queue.cc:260] T 00000000000000000000000000000000 P c819904cd64a46c1a615785c043c9c1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c819904cd64a46c1a615785c043c9c1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44179 } }
I20260810 23:30:54.415599 16430 raft_consensus.cc:388] T 00000000000000000000000000000000 P c819904cd64a46c1a615785c043c9c1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:30:54.415747 16430 raft_consensus.cc:482] T 00000000000000000000000000000000 P c819904cd64a46c1a615785c043c9c1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:30:54.415928 16430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c819904cd64a46c1a615785c043c9c1e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:30:54.416150 15037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:30:54.419204 16430 raft_consensus.cc:504] T 00000000000000000000000000000000 P c819904cd64a46c1a615785c043c9c1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c819904cd64a46c1a615785c043c9c1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44179 } }
I20260810 23:30:54.419915 16430 leader_election.cc:302] T 00000000000000000000000000000000 P c819904cd64a46c1a615785c043c9c1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c819904cd64a46c1a615785c043c9c1e; no voters:
I20260810 23:30:54.420655 16430 leader_election.cc:288] T 00000000000000000000000000000000 P c819904cd64a46c1a615785c043c9c1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:30:54.420771 16441 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c819904cd64a46c1a615785c043c9c1e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:30:54.425879 16441 raft_consensus.cc:686] T 00000000000000000000000000000000 P c819904cd64a46c1a615785c043c9c1e [term 1 LEADER]: Becoming Leader. State: Replica: c819904cd64a46c1a615785c043c9c1e, State: Running, Role: LEADER
I20260810 23:30:54.426687 16441 consensus_queue.cc:237] T 00000000000000000000000000000000 P c819904cd64a46c1a615785c043c9c1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c819904cd64a46c1a615785c043c9c1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44179 } }
I20260810 23:30:54.427337 16430 sys_catalog.cc:564] T 00000000000000000000000000000000 P c819904cd64a46c1a615785c043c9c1e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:30:54.429616 16443 sys_catalog.cc:455] T 00000000000000000000000000000000 P c819904cd64a46c1a615785c043c9c1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c819904cd64a46c1a615785c043c9c1e. Latest consensus state: current_term: 1 leader_uuid: "c819904cd64a46c1a615785c043c9c1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c819904cd64a46c1a615785c043c9c1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44179 } } }
I20260810 23:30:54.430586 16443 sys_catalog.cc:458] T 00000000000000000000000000000000 P c819904cd64a46c1a615785c043c9c1e [sys.catalog]: This master's current role is: LEADER
I20260810 23:30:54.430280 16442 sys_catalog.cc:455] T 00000000000000000000000000000000 P c819904cd64a46c1a615785c043c9c1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c819904cd64a46c1a615785c043c9c1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c819904cd64a46c1a615785c043c9c1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44179 } } }
I20260810 23:30:54.431140 16442 sys_catalog.cc:458] T 00000000000000000000000000000000 P c819904cd64a46c1a615785c043c9c1e [sys.catalog]: This master's current role is: LEADER
I20260810 23:30:54.446553 16452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:30:54.447203 16433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:54.421836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:49725 (local address 127.0.0.1:44179)
0810 23:30:54.422034 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:54.422041 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:54.426696 (+ 4655us) server_negotiation.cc:407] Connection header received
0810 23:30:54.427103 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:54.427110 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:54.427198 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:54.427276 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:54.429721 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:54.431116 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:54.433567 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:54.434409 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:54.434536 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:54.438638 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:54.438646 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:54.438652 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:30:54.438709 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:54.442652 (+ 3943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:54.443940 (+ 1288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:54.443947 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:54.443951 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:54.444132 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:30:54.444808 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:30:54.444814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:30:54.444818 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:30:54.445350 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:54.445685 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:54.446622 (+ 937us) server_negotiation.cc:299] Negotiation successful
0810 23:30:54.446830 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 23:30:54.452661 16452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:30:54.462371 16452 catalog_manager.cc:1347] Generated new cluster ID: 0ba107df5a1845fe9e15a50154e09fb7
I20260810 23:30:54.462500 16452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:30:54.473318 16452 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:30:54.475410 16452 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:30:54.485273 16452 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c819904cd64a46c1a615785c043c9c1e: Generated new TSK 0
I20260810 23:30:54.486274 16452 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:30:54.497931 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 16322
I20260810 23:30:54.499609 16423 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:30:54.593060 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 16322
W20260810 23:30:54.618112 16294 connection.cc:535] client connection to 127.0.0.1:44179 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44179): Connection reset by peer (error 104)
2026-08-10T23:30:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2263 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:30:54 dist-test-slave-rmhg krb5kdc[16476](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:30:54 dist-test-slave-rmhg krb5kdc[16476](info): set up 2 sockets
Aug 10 23:30:54 dist-test-slave-rmhg krb5kdc[16476](info): commencing operation
krb5kdc: starting...
W20260810 23:30:55.858779 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.221s user 0.003s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:30:56 dist-test-slave-rmhg krb5kdc[16476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:30:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:30:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:30:56.197178 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46365
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:30:56.556840 16499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:30:56.557212 16499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:30:56.557368 16499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:30:56.565028 16499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:30:56.565202 16499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:30:56.565294 16499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:30:56.565378 16499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:30:56.565439 16499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:30:56.575927 16499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:30:56.577447 16499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:30:56.578904 16499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:30:56.591657 16507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:56.591794 16508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:56.591815 16512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:30:56.592480 16499 server_base.cc:1034] running on GCE node
I20260810 23:30:56.593361 16499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:30:56.594888 16499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:30:56.596113 16499 hybrid_clock.cc:648] HybridClock initialized: now 1786404656596029 us; error 58 us; skew 500 ppm
I20260810 23:30:56.596444 16499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:30:56 dist-test-slave-rmhg krb5kdc[16476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:30:56.599695 16499 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:30:56.601339 16499 webserver.cc:460] Webserver started at http://127.0.0.1:45341/ using document root <none> and password file <none>
I20260810 23:30:56.602041 16499 fs_manager.cc:359] Metadata directory not provided
I20260810 23:30:56.602160 16499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:30:56.602434 16499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:30:56.605659 16499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "19018d0d491542d3bcc79d1c843d1992"
format_stamp: "Formatted at 2026-08-10 23:30:56 on dist-test-slave-rmhg"
server_key: "18c67b247de5f9809267c8f853746a90"
server_key_iv: "ac77570ed58725e70b418f71e65cb4ad"
server_key_version: "encryptionkey@0"
I20260810 23:30:56.606426 16499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "19018d0d491542d3bcc79d1c843d1992"
format_stamp: "Formatted at 2026-08-10 23:30:56 on dist-test-slave-rmhg"
server_key: "18c67b247de5f9809267c8f853746a90"
server_key_iv: "ac77570ed58725e70b418f71e65cb4ad"
server_key_version: "encryptionkey@0"
I20260810 23:30:56.612038 16499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:30:56.615832 16523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:56.617336 16499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:30:56.617489 16499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "19018d0d491542d3bcc79d1c843d1992"
format_stamp: "Formatted at 2026-08-10 23:30:56 on dist-test-slave-rmhg"
server_key: "18c67b247de5f9809267c8f853746a90"
server_key_iv: "ac77570ed58725e70b418f71e65cb4ad"
server_key_version: "encryptionkey@0"
I20260810 23:30:56.617679 16499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:30:56.636277 16499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:30:56.640465 16499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:30:56.640719 16499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:30:56.659291 16499 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46365
I20260810 23:30:56.659317 16601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46365 every 8 connection(s)
I20260810 23:30:56.660766 16499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:30:56.661051 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 16499
I20260810 23:30:56.661264 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:30:56.661868 15037 external_mini_cluster.cc:1428] Setting key 32ec510e57cfd3aab84de2d2795e40ba
I20260810 23:30:56.665604 16602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:30:56.675459 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:56.663814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:49063 (local address 127.0.0.1:46365)
0810 23:30:56.664632 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:56.664648 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:56.664706 (+ 58us) server_negotiation.cc:407] Connection header received
0810 23:30:56.666448 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:56.666473 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:56.666891 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:56.667248 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:56.667858 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:56.669357 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:56.670818 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:56.671544 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:56.671729 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:56.673005 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:56.673013 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:56.673030 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:56.673665 (+ 635us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:56.673745 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:56.673785 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:56.673934 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:56.674256 (+ 322us) server_negotiation.cc:299] Negotiation successful
0810 23:30:56.674423 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":313,"threads_started":1}
I20260810 23:30:56.677222 16602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19018d0d491542d3bcc79d1c843d1992: Bootstrap starting.
I20260810 23:30:56.679641 15037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:30:56.680606 16602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19018d0d491542d3bcc79d1c843d1992: Neither blocks nor log segments found. Creating new log.
I20260810 23:30:56.681701 16602 log.cc:824] T 00000000000000000000000000000000 P 19018d0d491542d3bcc79d1c843d1992: Log is configured to *not* fsync() on all Append() calls
I20260810 23:30:56.684870 16602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19018d0d491542d3bcc79d1c843d1992: No bootstrap required, opened a new log
I20260810 23:30:56.691129 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:56.684022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:41455 (local address 127.0.0.1:46365)
0810 23:30:56.684213 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:56.684221 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:56.684786 (+ 565us) server_negotiation.cc:407] Connection header received
0810 23:30:56.684999 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:56.685004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:56.685075 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:56.685188 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:56.685815 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:56.686608 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:56.688482 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:30:56.688972 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:30:56.689073 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:30:56.690267 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:56.690272 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:56.690275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:56.690296 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:56.690336 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:56.690352 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:56.690439 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:56.690725 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 23:30:56.690783 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 23:30:56.694514 16602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19018d0d491542d3bcc79d1c843d1992 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19018d0d491542d3bcc79d1c843d1992" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46365 } }
I20260810 23:30:56.694970 16602 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19018d0d491542d3bcc79d1c843d1992 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:30:56.695052 16602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19018d0d491542d3bcc79d1c843d1992 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19018d0d491542d3bcc79d1c843d1992, State: Initialized, Role: FOLLOWER
I20260810 23:30:56.695597 16602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19018d0d491542d3bcc79d1c843d1992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19018d0d491542d3bcc79d1c843d1992" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46365 } }
I20260810 23:30:56.695801 16602 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19018d0d491542d3bcc79d1c843d1992 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:30:56.695928 16602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19018d0d491542d3bcc79d1c843d1992 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:30:56.696053 16602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19018d0d491542d3bcc79d1c843d1992 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:30:56.698284 16602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19018d0d491542d3bcc79d1c843d1992 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19018d0d491542d3bcc79d1c843d1992" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46365 } }
I20260810 23:30:56.698770 16602 leader_election.cc:302] T 00000000000000000000000000000000 P 19018d0d491542d3bcc79d1c843d1992 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19018d0d491542d3bcc79d1c843d1992; no voters:
I20260810 23:30:56.699366 16602 leader_election.cc:288] T 00000000000000000000000000000000 P 19018d0d491542d3bcc79d1c843d1992 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:30:56.699512 16617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19018d0d491542d3bcc79d1c843d1992 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:30:56.700206 16617 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19018d0d491542d3bcc79d1c843d1992 [term 1 LEADER]: Becoming Leader. State: Replica: 19018d0d491542d3bcc79d1c843d1992, State: Running, Role: LEADER
I20260810 23:30:56.700832 16617 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19018d0d491542d3bcc79d1c843d1992 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19018d0d491542d3bcc79d1c843d1992" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46365 } }
I20260810 23:30:56.701752 16602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19018d0d491542d3bcc79d1c843d1992 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:30:56.704193 16621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19018d0d491542d3bcc79d1c843d1992 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19018d0d491542d3bcc79d1c843d1992. Latest consensus state: current_term: 1 leader_uuid: "19018d0d491542d3bcc79d1c843d1992" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19018d0d491542d3bcc79d1c843d1992" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46365 } } }
I20260810 23:30:56.704700 16621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19018d0d491542d3bcc79d1c843d1992 [sys.catalog]: This master's current role is: LEADER
I20260810 23:30:56.705291 16618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19018d0d491542d3bcc79d1c843d1992 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19018d0d491542d3bcc79d1c843d1992" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19018d0d491542d3bcc79d1c843d1992" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46365 } } }
I20260810 23:30:56.705585 16618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19018d0d491542d3bcc79d1c843d1992 [sys.catalog]: This master's current role is: LEADER
I20260810 23:30:56.709815 16637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:30:56.714811 16637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:30:56.724396 16637 catalog_manager.cc:1347] Generated new cluster ID: 6d0f8248d5f441e8b2d65bd6d95481cc
I20260810 23:30:56.724572 16637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:30:56.743898 16637 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:30:56.746038 16637 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:30:56.763370 16637 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19018d0d491542d3bcc79d1c843d1992: Generated new TSK 0
I20260810 23:30:56.764333 16637 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:30:56.777539 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 16499
W20260810 23:30:56.778995 16597 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.204:49063
W20260810 23:30:56.779853 15037 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 16499: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:30:56.780042 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 16499
W20260810 23:30:56.812439 16474 connection.cc:535] client connection to 127.0.0.1:46365 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46365): Connection reset by peer (error 104)
2026-08-10T23:30:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2194 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:30:56 dist-test-slave-rmhg krb5kdc[16656](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:30:56 dist-test-slave-rmhg krb5kdc[16656](info): set up 2 sockets
Aug 10 23:30:56 dist-test-slave-rmhg krb5kdc[16656](info): commencing operation
krb5kdc: starting...
W20260810 23:30:58.047576 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.216s user 0.005s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:30:58 dist-test-slave-rmhg krb5kdc[16656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:30:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:30:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:30:58.393266 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46441
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:30:58.746743 16676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:30:58.747062 16676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:30:58.747159 16676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:30:58.754699 16676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:30:58.754832 16676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:30:58.754889 16676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:30:58.754953 16676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:30:58.754998 16676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:30:58.765733 16676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:30:58.767102 16676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:30:58.768491 16676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:30:58.781385 16686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:58.781332 16688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:30:58.781332 16690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:30:58.782226 16676 server_base.cc:1034] running on GCE node
I20260810 23:30:58.783063 16676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:30:58.784655 16676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:30:58.785907 16676 hybrid_clock.cc:648] HybridClock initialized: now 1786404658785862 us; error 44 us; skew 500 ppm
I20260810 23:30:58.786335 16676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:30:58 dist-test-slave-rmhg krb5kdc[16656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:30:58.789748 16676 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:30:58.791512 16676 webserver.cc:460] Webserver started at http://127.0.0.1:39041/ using document root <none> and password file <none>
I20260810 23:30:58.792290 16676 fs_manager.cc:359] Metadata directory not provided
I20260810 23:30:58.792416 16676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:30:58.792745 16676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:30:58.796126 16676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "72400718454f4d9aada51f9090f7a368"
format_stamp: "Formatted at 2026-08-10 23:30:58 on dist-test-slave-rmhg"
server_key: "f910b2ad40f347d6548e97256148d088"
server_key_iv: "5dec6a4c26950e51e89a20eb9be4af4d"
server_key_version: "encryptionkey@0"
I20260810 23:30:58.797024 16676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "72400718454f4d9aada51f9090f7a368"
format_stamp: "Formatted at 2026-08-10 23:30:58 on dist-test-slave-rmhg"
server_key: "f910b2ad40f347d6548e97256148d088"
server_key_iv: "5dec6a4c26950e51e89a20eb9be4af4d"
server_key_version: "encryptionkey@0"
I20260810 23:30:58.803105 16676 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 23:30:58.807372 16700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:30:58.808971 16676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:30:58.809276 16676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "72400718454f4d9aada51f9090f7a368"
format_stamp: "Formatted at 2026-08-10 23:30:58 on dist-test-slave-rmhg"
server_key: "f910b2ad40f347d6548e97256148d088"
server_key_iv: "5dec6a4c26950e51e89a20eb9be4af4d"
server_key_version: "encryptionkey@0"
I20260810 23:30:58.809515 16676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:30:58.815974 16676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:30:58.820494 16676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:30:58.820848 16676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:30:58.840297 16676 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46441
I20260810 23:30:58.840337 16785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46441 every 8 connection(s)
I20260810 23:30:58.841795 16676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:30:58.845876 16786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:30:58.846174 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 16676
I20260810 23:30:58.846338 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:30:58.846786 15037 external_mini_cluster.cc:1428] Setting key d33a98876ad96dfc7ea4bd0f4b62faa2
I20260810 23:30:58.856696 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:58.848779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:53775 (local address 127.0.0.1:46441)
0810 23:30:58.849702 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:58.849718 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:58.849772 (+ 54us) server_negotiation.cc:407] Connection header received
0810 23:30:58.852576 (+ 2804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:58.852613 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:58.853198 (+ 585us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:58.853713 (+ 515us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:58.854700 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:58.854708 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:58.854725 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:58.854742 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:58.854814 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:58.854847 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:58.855070 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:58.855206 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 23:30:58.855427 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":396,"threads_started":1}
I20260810 23:30:58.860914 15037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:30:58.861258 16786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72400718454f4d9aada51f9090f7a368: Bootstrap starting.
I20260810 23:30:58.864552 16786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72400718454f4d9aada51f9090f7a368: Neither blocks nor log segments found. Creating new log.
I20260810 23:30:58.865864 16786 log.cc:824] T 00000000000000000000000000000000 P 72400718454f4d9aada51f9090f7a368: Log is configured to *not* fsync() on all Append() calls
I20260810 23:30:58.868181 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:30:58.865455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:46651 (local address 127.0.0.1:46441)
0810 23:30:58.865697 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 23:30:58.865705 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:30:58.866248 (+ 543us) server_negotiation.cc:407] Connection header received
0810 23:30:58.866395 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:30:58.866401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:30:58.866470 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:30:58.866586 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:30:58.867540 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:30:58.867546 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:30:58.867551 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:30:58.867570 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:30:58.867617 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:30:58.867637 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:30:58.867750 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 23:30:58.867829 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 23:30:58.867897 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 23:30:58.869378 16786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72400718454f4d9aada51f9090f7a368: No bootstrap required, opened a new log
I20260810 23:30:58.877306 16786 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72400718454f4d9aada51f9090f7a368 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72400718454f4d9aada51f9090f7a368" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46441 } }
I20260810 23:30:58.877715 16786 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72400718454f4d9aada51f9090f7a368 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:30:58.877794 16786 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72400718454f4d9aada51f9090f7a368 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72400718454f4d9aada51f9090f7a368, State: Initialized, Role: FOLLOWER
I20260810 23:30:58.878336 16786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72400718454f4d9aada51f9090f7a368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72400718454f4d9aada51f9090f7a368" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46441 } }
I20260810 23:30:58.878546 16786 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72400718454f4d9aada51f9090f7a368 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:30:58.878660 16786 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72400718454f4d9aada51f9090f7a368 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:30:58.878793 16786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72400718454f4d9aada51f9090f7a368 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:30:58.881062 16786 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72400718454f4d9aada51f9090f7a368 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72400718454f4d9aada51f9090f7a368" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46441 } }
I20260810 23:30:58.881667 16786 leader_election.cc:302] T 00000000000000000000000000000000 P 72400718454f4d9aada51f9090f7a368 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72400718454f4d9aada51f9090f7a368; no voters:
I20260810 23:30:58.882361 16786 leader_election.cc:288] T 00000000000000000000000000000000 P 72400718454f4d9aada51f9090f7a368 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:30:58.882476 16801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72400718454f4d9aada51f9090f7a368 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:30:58.883251 16801 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72400718454f4d9aada51f9090f7a368 [term 1 LEADER]: Becoming Leader. State: Replica: 72400718454f4d9aada51f9090f7a368, State: Running, Role: LEADER
I20260810 23:30:58.883795 16801 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72400718454f4d9aada51f9090f7a368 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72400718454f4d9aada51f9090f7a368" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46441 } }
I20260810 23:30:58.884647 16786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72400718454f4d9aada51f9090f7a368 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:30:58.886665 16804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72400718454f4d9aada51f9090f7a368 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72400718454f4d9aada51f9090f7a368. Latest consensus state: current_term: 1 leader_uuid: "72400718454f4d9aada51f9090f7a368" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72400718454f4d9aada51f9090f7a368" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46441 } } }
I20260810 23:30:58.886668 16803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72400718454f4d9aada51f9090f7a368 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72400718454f4d9aada51f9090f7a368" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72400718454f4d9aada51f9090f7a368" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46441 } } }
I20260810 23:30:58.887151 16803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72400718454f4d9aada51f9090f7a368 [sys.catalog]: This master's current role is: LEADER
I20260810 23:30:58.887151 16804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72400718454f4d9aada51f9090f7a368 [sys.catalog]: This master's current role is: LEADER
I20260810 23:30:58.889089 16811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:30:58.894996 16811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:30:58.904737 16811 catalog_manager.cc:1347] Generated new cluster ID: 21847476667b4e04b4a9e13cb53a2175
I20260810 23:30:58.904953 16811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:30:58.926046 16811 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:30:58.926508 16811 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:30:58.936698 16811 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72400718454f4d9aada51f9090f7a368: Generated new TSK 0
I20260810 23:30:58.937651 16811 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:30:58.950325 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 16676
W20260810 23:30:58.951579 16780 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.204:53775
W20260810 23:30:58.952383 15037 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 16676: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:30:58.952515 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 16676
2026-08-10T23:30:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2163 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:30:59 dist-test-slave-rmhg krb5kdc[16835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:30:59 dist-test-slave-rmhg krb5kdc[16835](info): set up 2 sockets
Aug 10 23:30:59 dist-test-slave-rmhg krb5kdc[16835](info): commencing operation
krb5kdc: starting...
W20260810 23:31:00.212942 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.218s user 0.000s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:31:00 dist-test-slave-rmhg krb5kdc[16835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:31:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:31:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:31:00.555716 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35417
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:31:00.914817 16852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:31:00.915140 16852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:31:00.915261 16852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:31:00.922998 16852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:31:00.923146 16852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:31:00.923208 16852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:31:00.923297 16852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:31:00.923365 16852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:31:00.933909 16852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35417
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:31:00.935343 16852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:31:00.936757 16852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:31:00.949391 16861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:00.949352 16862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:00.949425 16866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:31:00.950213 16852 server_base.cc:1034] running on GCE node
I20260810 23:31:00.951066 16852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:31:00.952636 16852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:31:00.953922 16852 hybrid_clock.cc:648] HybridClock initialized: now 1786404660953858 us; error 51 us; skew 500 ppm
I20260810 23:31:00.954312 16852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:31:00 dist-test-slave-rmhg krb5kdc[16835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:31:00.961289 16852 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:31:00.963093 16852 webserver.cc:460] Webserver started at http://127.0.0.1:39583/ using document root <none> and password file <none>
I20260810 23:31:00.963843 16852 fs_manager.cc:359] Metadata directory not provided
I20260810 23:31:00.963970 16852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:31:00.964321 16852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:31:00.967659 16852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "004d4865a94d4955a8d184aea4267471"
format_stamp: "Formatted at 2026-08-10 23:31:00 on dist-test-slave-rmhg"
server_key: "ef71b73e995a383d286b0ba5909d1b84"
server_key_iv: "8ceb1cefdfc72c7b8e67e8b8877304b4"
server_key_version: "encryptionkey@0"
I20260810 23:31:00.968509 16852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "004d4865a94d4955a8d184aea4267471"
format_stamp: "Formatted at 2026-08-10 23:31:00 on dist-test-slave-rmhg"
server_key: "ef71b73e995a383d286b0ba5909d1b84"
server_key_iv: "8ceb1cefdfc72c7b8e67e8b8877304b4"
server_key_version: "encryptionkey@0"
I20260810 23:31:00.974231 16852 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 23:31:00.978066 16878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:00.979512 16852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:31:00.979699 16852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "004d4865a94d4955a8d184aea4267471"
format_stamp: "Formatted at 2026-08-10 23:31:00 on dist-test-slave-rmhg"
server_key: "ef71b73e995a383d286b0ba5909d1b84"
server_key_iv: "8ceb1cefdfc72c7b8e67e8b8877304b4"
server_key_version: "encryptionkey@0"
I20260810 23:31:00.979892 16852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:31:01.014510 16852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:31:01.019071 16852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:31:01.019358 16852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:31:01.037997 16852 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35417
I20260810 23:31:01.038020 16962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35417 every 8 connection(s)
I20260810 23:31:01.039494 16852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:31:01.040639 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 16852
I20260810 23:31:01.040841 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:31:01.041440 15037 external_mini_cluster.cc:1428] Setting key c55b9d14b37012170241218fbab731ae
I20260810 23:31:01.044811 16963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 23:31:01 dist-test-slave-rmhg krb5kdc[16835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 23:31:01.059544 16963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 004d4865a94d4955a8d184aea4267471: Bootstrap starting.
I20260810 23:31:01.063035 16963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 004d4865a94d4955a8d184aea4267471: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:01.064366 16963 log.cc:824] T 00000000000000000000000000000000 P 004d4865a94d4955a8d184aea4267471: Log is configured to *not* fsync() on all Append() calls
I20260810 23:31:01.067502 16966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:01.043485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:55181 (local address 127.0.0.1:35417)
0810 23:31:01.044376 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:01.044389 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:01.044432 (+ 43us) server_negotiation.cc:407] Connection header received
0810 23:31:01.046251 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:01.046277 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:01.046737 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:01.047101 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:01.049235 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:01.050763 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:01.052691 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:01.053527 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:01.053686 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:01.059261 (+ 5575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:01.059272 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:01.059291 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:01.059341 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:01.062517 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:01.063615 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:01.063623 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:01.063630 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:01.063811 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:01.064344 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:01.064347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:01.064353 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:01.064977 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:01.065366 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:01.065812 (+ 446us) server_negotiation.cc:299] Negotiation successful
0810 23:31:01.066166 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":416,"threads_started":1}
I20260810 23:31:01.068014 16963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 004d4865a94d4955a8d184aea4267471: No bootstrap required, opened a new log
I20260810 23:31:01.071815 15037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:31:01.076445 16963 raft_consensus.cc:348] T 00000000000000000000000000000000 P 004d4865a94d4955a8d184aea4267471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004d4865a94d4955a8d184aea4267471" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35417 } }
I20260810 23:31:01.076876 16963 raft_consensus.cc:374] T 00000000000000000000000000000000 P 004d4865a94d4955a8d184aea4267471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:01.076972 16963 raft_consensus.cc:729] T 00000000000000000000000000000000 P 004d4865a94d4955a8d184aea4267471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 004d4865a94d4955a8d184aea4267471, State: Initialized, Role: FOLLOWER
I20260810 23:31:01.077641 16963 consensus_queue.cc:260] T 00000000000000000000000000000000 P 004d4865a94d4955a8d184aea4267471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004d4865a94d4955a8d184aea4267471" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35417 } }
I20260810 23:31:01.077874 16963 raft_consensus.cc:388] T 00000000000000000000000000000000 P 004d4865a94d4955a8d184aea4267471 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:31:01.078011 16963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 004d4865a94d4955a8d184aea4267471 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:31:01.078167 16963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 004d4865a94d4955a8d184aea4267471 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:01.080977 16963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 004d4865a94d4955a8d184aea4267471 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004d4865a94d4955a8d184aea4267471" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35417 } }
I20260810 23:31:01.081537 16963 leader_election.cc:302] T 00000000000000000000000000000000 P 004d4865a94d4955a8d184aea4267471 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 004d4865a94d4955a8d184aea4267471; no voters:
I20260810 23:31:01.082249 16963 leader_election.cc:288] T 00000000000000000000000000000000 P 004d4865a94d4955a8d184aea4267471 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:31:01.082387 16974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 004d4865a94d4955a8d184aea4267471 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:31:01.084664 16963 sys_catalog.cc:564] T 00000000000000000000000000000000 P 004d4865a94d4955a8d184aea4267471 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:31:01.085868 16974 raft_consensus.cc:686] T 00000000000000000000000000000000 P 004d4865a94d4955a8d184aea4267471 [term 1 LEADER]: Becoming Leader. State: Replica: 004d4865a94d4955a8d184aea4267471, State: Running, Role: LEADER
I20260810 23:31:01.086555 16974 consensus_queue.cc:237] T 00000000000000000000000000000000 P 004d4865a94d4955a8d184aea4267471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004d4865a94d4955a8d184aea4267471" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35417 } }
I20260810 23:31:01.089638 16977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 004d4865a94d4955a8d184aea4267471 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 004d4865a94d4955a8d184aea4267471. Latest consensus state: current_term: 1 leader_uuid: "004d4865a94d4955a8d184aea4267471" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004d4865a94d4955a8d184aea4267471" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35417 } } }
I20260810 23:31:01.090103 16977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 004d4865a94d4955a8d184aea4267471 [sys.catalog]: This master's current role is: LEADER
I20260810 23:31:01.089973 16976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 004d4865a94d4955a8d184aea4267471 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "004d4865a94d4955a8d184aea4267471" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004d4865a94d4955a8d184aea4267471" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35417 } } }
I20260810 23:31:01.090422 16976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 004d4865a94d4955a8d184aea4267471 [sys.catalog]: This master's current role is: LEADER
I20260810 23:31:01.094506 16966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:01.076304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:44263 (local address 127.0.0.1:35417)
0810 23:31:01.076482 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:01.076488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:01.076837 (+ 349us) server_negotiation.cc:407] Connection header received
0810 23:31:01.077035 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:01.077040 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:01.077125 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:01.077257 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:01.078885 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:01.080077 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:01.081437 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:01.082122 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:01.082287 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:01.085792 (+ 3505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:01.085801 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:01.085806 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:01.085864 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:01.090133 (+ 4269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:01.091364 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:01.091370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:01.091373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:01.091570 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:01.092239 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:01.092244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:01.092247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:01.092693 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:01.093062 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:01.093955 (+ 893us) server_negotiation.cc:299] Negotiation successful
0810 23:31:01.094140 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260810 23:31:01.098800 16994 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 004d4865a94d4955a8d184aea4267471: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:31:01.098920 16994 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:31:01.099017 16985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:31:01.101074 16985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:31:01.110440 16985 catalog_manager.cc:1347] Generated new cluster ID: 5eb4cb5ce9ac4047bac95680271e6735
I20260810 23:31:01.110584 16985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:31:01.124291 16985 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:31:01.125780 16985 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:31:01.135957 16985 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 004d4865a94d4955a8d184aea4267471: Generated new TSK 0
I20260810 23:31:01.137027 16985 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:31:01.152756 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 16852
I20260810 23:31:01.154016 16954 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:31:01.248728 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 16852
W20260810 23:31:01.276041 16833 connection.cc:535] client connection to 127.0.0.1:35417 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35417): Connection reset by peer (error 104)
2026-08-10T23:31:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:31:01 dist-test-slave-rmhg krb5kdc[17006](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:31:01 dist-test-slave-rmhg krb5kdc[17006](info): set up 2 sockets
Aug 10 23:31:01 dist-test-slave-rmhg krb5kdc[17006](info): commencing operation
krb5kdc: starting...
W20260810 23:31:02.516745 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.221s user 0.002s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:31:02 dist-test-slave-rmhg krb5kdc[17006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:31:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:31:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:31:02.854255 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45351
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:31:03.209395 17027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:31:03.209714 17027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:31:03.209856 17027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:31:03.217347 17027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:31:03.217486 17027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:31:03.217561 17027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:31:03.217656 17027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:31:03.217718 17027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:31:03.228542 17027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45351
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:31:03.230019 17027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:31:03.231384 17027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:31:03.243551 17042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:03.243727 17035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:03.243705 17036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:31:03.244956 17027 server_base.cc:1034] running on GCE node
I20260810 23:31:03.245863 17027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:31:03.247494 17027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:31:03.248749 17027 hybrid_clock.cc:648] HybridClock initialized: now 1786404663248701 us; error 72 us; skew 500 ppm
I20260810 23:31:03.249145 17027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:31:03 dist-test-slave-rmhg krb5kdc[17006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:31:03.252492 17027 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:31:03.254216 17027 webserver.cc:460] Webserver started at http://127.0.0.1:33705/ using document root <none> and password file <none>
I20260810 23:31:03.254954 17027 fs_manager.cc:359] Metadata directory not provided
I20260810 23:31:03.255087 17027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:31:03.255400 17027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:31:03.258625 17027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "38de49ca6d5a478196ec8a3cca407a7a"
format_stamp: "Formatted at 2026-08-10 23:31:03 on dist-test-slave-rmhg"
server_key: "29e559a7c28e81c6fea772590dfe1b1e"
server_key_iv: "34860775843b7a0078ba1f4abc6a3a1c"
server_key_version: "encryptionkey@0"
I20260810 23:31:03.259421 17027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "38de49ca6d5a478196ec8a3cca407a7a"
format_stamp: "Formatted at 2026-08-10 23:31:03 on dist-test-slave-rmhg"
server_key: "29e559a7c28e81c6fea772590dfe1b1e"
server_key_iv: "34860775843b7a0078ba1f4abc6a3a1c"
server_key_version: "encryptionkey@0"
I20260810 23:31:03.265017 17027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 23:31:03.269511 17052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:03.270993 17027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:31:03.271173 17027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "38de49ca6d5a478196ec8a3cca407a7a"
format_stamp: "Formatted at 2026-08-10 23:31:03 on dist-test-slave-rmhg"
server_key: "29e559a7c28e81c6fea772590dfe1b1e"
server_key_iv: "34860775843b7a0078ba1f4abc6a3a1c"
server_key_version: "encryptionkey@0"
I20260810 23:31:03.271375 17027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:31:03.302537 17027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:31:03.307238 17027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:31:03.307538 17027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:31:03.326400 17027 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45351
I20260810 23:31:03.326421 17134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45351 every 8 connection(s)
I20260810 23:31:03.327831 17027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:31:03.328655 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 17027
I20260810 23:31:03.328864 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:31:03.329440 15037 external_mini_cluster.cc:1428] Setting key 03cf738de8a4abecd48d587327d43134
I20260810 23:31:03.332381 17135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:03.343098 17138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:03.331320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:40435 (local address 127.0.0.1:45351)
0810 23:31:03.332116 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:03.332131 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:03.332183 (+ 52us) server_negotiation.cc:407] Connection header received
0810 23:31:03.333952 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:03.333980 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:03.334408 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:03.334791 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:03.335408 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:03.336795 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:03.338182 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:03.338904 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:03.339081 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:03.340425 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:03.340434 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:03.340455 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:31:03.341086 (+ 631us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:03.341224 (+ 138us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:31:03.341260 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:03.341453 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:03.341785 (+ 332us) server_negotiation.cc:299] Negotiation successful
0810 23:31:03.342008 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":339,"threads_started":1}
I20260810 23:31:03.344300 17135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38de49ca6d5a478196ec8a3cca407a7a: Bootstrap starting.
I20260810 23:31:03.347419 15037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:31:03.348026 17135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38de49ca6d5a478196ec8a3cca407a7a: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:03.349216 17135 log.cc:824] T 00000000000000000000000000000000 P 38de49ca6d5a478196ec8a3cca407a7a: Log is configured to *not* fsync() on all Append() calls
I20260810 23:31:03.353029 17135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38de49ca6d5a478196ec8a3cca407a7a: No bootstrap required, opened a new log
I20260810 23:31:03.358482 17138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:03.352140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:39855 (local address 127.0.0.1:45351)
0810 23:31:03.352296 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:03.352301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:03.352836 (+ 535us) server_negotiation.cc:407] Connection header received
0810 23:31:03.353037 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:03.353041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:03.353105 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:03.353284 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:03.353817 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:03.354604 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:03.355967 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:03.356450 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:03.356562 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:03.357771 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:03.357776 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:03.357779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:31:03.357800 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:03.357841 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:31:03.357856 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:03.357958 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:03.358151 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 23:31:03.358208 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:31:03.361411 17135 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38de49ca6d5a478196ec8a3cca407a7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38de49ca6d5a478196ec8a3cca407a7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45351 } }
I20260810 23:31:03.361814 17135 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38de49ca6d5a478196ec8a3cca407a7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:03.361908 17135 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38de49ca6d5a478196ec8a3cca407a7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38de49ca6d5a478196ec8a3cca407a7a, State: Initialized, Role: FOLLOWER
I20260810 23:31:03.362542 17135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38de49ca6d5a478196ec8a3cca407a7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38de49ca6d5a478196ec8a3cca407a7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45351 } }
I20260810 23:31:03.362859 17135 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38de49ca6d5a478196ec8a3cca407a7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:31:03.363031 17135 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38de49ca6d5a478196ec8a3cca407a7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:31:03.363247 17135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38de49ca6d5a478196ec8a3cca407a7a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:03.365694 17135 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38de49ca6d5a478196ec8a3cca407a7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38de49ca6d5a478196ec8a3cca407a7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45351 } }
I20260810 23:31:03.366220 17135 leader_election.cc:302] T 00000000000000000000000000000000 P 38de49ca6d5a478196ec8a3cca407a7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38de49ca6d5a478196ec8a3cca407a7a; no voters:
I20260810 23:31:03.366760 17135 leader_election.cc:288] T 00000000000000000000000000000000 P 38de49ca6d5a478196ec8a3cca407a7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:31:03.366921 17147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38de49ca6d5a478196ec8a3cca407a7a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:31:03.367560 17147 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38de49ca6d5a478196ec8a3cca407a7a [term 1 LEADER]: Becoming Leader. State: Replica: 38de49ca6d5a478196ec8a3cca407a7a, State: Running, Role: LEADER
I20260810 23:31:03.368093 17147 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38de49ca6d5a478196ec8a3cca407a7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38de49ca6d5a478196ec8a3cca407a7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45351 } }
I20260810 23:31:03.368831 17135 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38de49ca6d5a478196ec8a3cca407a7a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:31:03.371061 17150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38de49ca6d5a478196ec8a3cca407a7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38de49ca6d5a478196ec8a3cca407a7a. Latest consensus state: current_term: 1 leader_uuid: "38de49ca6d5a478196ec8a3cca407a7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38de49ca6d5a478196ec8a3cca407a7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45351 } } }
I20260810 23:31:03.371495 17150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38de49ca6d5a478196ec8a3cca407a7a [sys.catalog]: This master's current role is: LEADER
I20260810 23:31:03.371989 17149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38de49ca6d5a478196ec8a3cca407a7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38de49ca6d5a478196ec8a3cca407a7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38de49ca6d5a478196ec8a3cca407a7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45351 } } }
I20260810 23:31:03.372591 17149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38de49ca6d5a478196ec8a3cca407a7a [sys.catalog]: This master's current role is: LEADER
I20260810 23:31:03.373975 17154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:31:03.379539 17154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:31:03.389110 17154 catalog_manager.cc:1347] Generated new cluster ID: faa3bd08fe1c482ca5b1964722f7158a
I20260810 23:31:03.389257 17154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:31:03.408495 17154 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:31:03.409942 17154 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:31:03.419324 17154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38de49ca6d5a478196ec8a3cca407a7a: Generated new TSK 0
I20260810 23:31:03.420563 17154 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:31:03.438278 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 17027
W20260810 23:31:03.439567 17130 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.204:40435
W20260810 23:31:03.440438 15037 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 17027: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:31:03.440613 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 17027
W20260810 23:31:03.467146 17004 connection.cc:535] client connection to 127.0.0.1:45351 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45351): Connection reset by peer (error 104)
2026-08-10T23:31:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2190 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:31:03 dist-test-slave-rmhg krb5kdc[17191](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:31:03 dist-test-slave-rmhg krb5kdc[17191](info): set up 2 sockets
Aug 10 23:31:03 dist-test-slave-rmhg krb5kdc[17191](info): commencing operation
krb5kdc: starting...
W20260810 23:31:04.700232 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.214s user 0.001s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:31:04 dist-test-slave-rmhg krb5kdc[17191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:31:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:31:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:31:05.054678 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37725
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:31:05.424573 17215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:31:05.424909 17215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:31:05.425012 17215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:31:05.432545 17215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:31:05.432686 17215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:31:05.432742 17215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:31:05.432807 17215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:31:05.432853 17215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:31:05.443208 17215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37725
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404299514076-15037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:31:05.444644 17215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:31:05.446110 17215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:31:05.459038 17230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:05.459044 17225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:05.459081 17224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:31:05.460292 17215 server_base.cc:1034] running on GCE node
I20260810 23:31:05.461187 17215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:31:05.462836 17215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:31:05.464051 17215 hybrid_clock.cc:648] HybridClock initialized: now 1786404665464004 us; error 48 us; skew 500 ppm
I20260810 23:31:05.464421 17215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:31:05 dist-test-slave-rmhg krb5kdc[17191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:31:05.467826 17215 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:31:05.469447 17215 webserver.cc:460] Webserver started at http://127.0.0.1:43811/ using document root <none> and password file <none>
I20260810 23:31:05.470176 17215 fs_manager.cc:359] Metadata directory not provided
I20260810 23:31:05.470302 17215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:31:05.470618 17215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:31:05.473860 17215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "f95d40525cd84ef6b9c566db79bc6210"
format_stamp: "Formatted at 2026-08-10 23:31:05 on dist-test-slave-rmhg"
server_key: "d53749714d56ee298f75bea52bcad931"
server_key_iv: "20f3c184ffc3d1abb2731544a2e1fa0f"
server_key_version: "encryptionkey@0"
I20260810 23:31:05.474656 17215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "f95d40525cd84ef6b9c566db79bc6210"
format_stamp: "Formatted at 2026-08-10 23:31:05 on dist-test-slave-rmhg"
server_key: "d53749714d56ee298f75bea52bcad931"
server_key_iv: "20f3c184ffc3d1abb2731544a2e1fa0f"
server_key_version: "encryptionkey@0"
I20260810 23:31:05.480520 17215 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 23:31:05.484473 17247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:05.486037 17215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:31:05.486231 17215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "f95d40525cd84ef6b9c566db79bc6210"
format_stamp: "Formatted at 2026-08-10 23:31:05 on dist-test-slave-rmhg"
server_key: "d53749714d56ee298f75bea52bcad931"
server_key_iv: "20f3c184ffc3d1abb2731544a2e1fa0f"
server_key_version: "encryptionkey@0"
I20260810 23:31:05.486433 17215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:31:05.492514 17215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:31:05.496652 17215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:31:05.496953 17215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:31:05.515419 17215 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37725
I20260810 23:31:05.515468 17338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37725 every 8 connection(s)
I20260810 23:31:05.516952 17215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:31:05.518441 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 17215
I20260810 23:31:05.518617 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:31:05.519039 15037 external_mini_cluster.cc:1428] Setting key ff1d635b677cc403a55f948f01e0f31b
I20260810 23:31:05.521672 17339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:05.527320 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:05.520942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:36093 (local address 127.0.0.1:37725)
0810 23:31:05.521830 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:05.521847 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:05.521899 (+ 52us) server_negotiation.cc:407] Connection header received
0810 23:31:05.523562 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:05.523588 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:05.523969 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:05.524345 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:05.525562 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:05.525569 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:05.525584 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:31:05.525631 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:05.525725 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:31:05.525750 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:05.525937 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:05.526038 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 23:31:05.526250 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":380,"threads_started":1}
I20260810 23:31:05.531742 15037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:31:05.534687 17339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f95d40525cd84ef6b9c566db79bc6210: Bootstrap starting.
I20260810 23:31:05.538892 17339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f95d40525cd84ef6b9c566db79bc6210: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:05.539193 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:05.536573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:56687 (local address 127.0.0.1:37725)
0810 23:31:05.536776 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:05.536783 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:05.537067 (+ 284us) server_negotiation.cc:407] Connection header received
0810 23:31:05.537279 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:05.537283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:05.537337 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:05.537423 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:05.538432 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:05.538436 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:05.538439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:31:05.538455 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:05.538524 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:31:05.538541 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:05.538634 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:05.538821 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 23:31:05.538881 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 23:31:05.540318 17339 log.cc:824] T 00000000000000000000000000000000 P f95d40525cd84ef6b9c566db79bc6210: Log is configured to *not* fsync() on all Append() calls
I20260810 23:31:05.543321 17339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f95d40525cd84ef6b9c566db79bc6210: No bootstrap required, opened a new log
I20260810 23:31:05.551012 17339 raft_consensus.cc:348] T 00000000000000000000000000000000 P f95d40525cd84ef6b9c566db79bc6210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95d40525cd84ef6b9c566db79bc6210" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37725 } }
I20260810 23:31:05.551498 17339 raft_consensus.cc:374] T 00000000000000000000000000000000 P f95d40525cd84ef6b9c566db79bc6210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:05.551601 17339 raft_consensus.cc:729] T 00000000000000000000000000000000 P f95d40525cd84ef6b9c566db79bc6210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f95d40525cd84ef6b9c566db79bc6210, State: Initialized, Role: FOLLOWER
I20260810 23:31:05.552268 17339 consensus_queue.cc:260] T 00000000000000000000000000000000 P f95d40525cd84ef6b9c566db79bc6210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95d40525cd84ef6b9c566db79bc6210" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37725 } }
I20260810 23:31:05.552522 17339 raft_consensus.cc:388] T 00000000000000000000000000000000 P f95d40525cd84ef6b9c566db79bc6210 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:31:05.552667 17339 raft_consensus.cc:482] T 00000000000000000000000000000000 P f95d40525cd84ef6b9c566db79bc6210 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:31:05.552834 17339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f95d40525cd84ef6b9c566db79bc6210 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:05.555277 17339 raft_consensus.cc:504] T 00000000000000000000000000000000 P f95d40525cd84ef6b9c566db79bc6210 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95d40525cd84ef6b9c566db79bc6210" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37725 } }
I20260810 23:31:05.555850 17339 leader_election.cc:302] T 00000000000000000000000000000000 P f95d40525cd84ef6b9c566db79bc6210 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f95d40525cd84ef6b9c566db79bc6210; no voters:
I20260810 23:31:05.556471 17339 leader_election.cc:288] T 00000000000000000000000000000000 P f95d40525cd84ef6b9c566db79bc6210 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:31:05.556610 17357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f95d40525cd84ef6b9c566db79bc6210 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:31:05.557571 17357 raft_consensus.cc:686] T 00000000000000000000000000000000 P f95d40525cd84ef6b9c566db79bc6210 [term 1 LEADER]: Becoming Leader. State: Replica: f95d40525cd84ef6b9c566db79bc6210, State: Running, Role: LEADER
I20260810 23:31:05.558377 17357 consensus_queue.cc:237] T 00000000000000000000000000000000 P f95d40525cd84ef6b9c566db79bc6210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95d40525cd84ef6b9c566db79bc6210" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37725 } }
I20260810 23:31:05.558717 17339 sys_catalog.cc:564] T 00000000000000000000000000000000 P f95d40525cd84ef6b9c566db79bc6210 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:31:05.561772 17358 sys_catalog.cc:455] T 00000000000000000000000000000000 P f95d40525cd84ef6b9c566db79bc6210 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f95d40525cd84ef6b9c566db79bc6210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95d40525cd84ef6b9c566db79bc6210" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37725 } } }
I20260810 23:31:05.561800 17359 sys_catalog.cc:455] T 00000000000000000000000000000000 P f95d40525cd84ef6b9c566db79bc6210 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f95d40525cd84ef6b9c566db79bc6210. Latest consensus state: current_term: 1 leader_uuid: "f95d40525cd84ef6b9c566db79bc6210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95d40525cd84ef6b9c566db79bc6210" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37725 } } }
I20260810 23:31:05.562264 17358 sys_catalog.cc:458] T 00000000000000000000000000000000 P f95d40525cd84ef6b9c566db79bc6210 [sys.catalog]: This master's current role is: LEADER
I20260810 23:31:05.562333 17359 sys_catalog.cc:458] T 00000000000000000000000000000000 P f95d40525cd84ef6b9c566db79bc6210 [sys.catalog]: This master's current role is: LEADER
I20260810 23:31:05.563818 17371 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:31:05.568830 17371 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:31:05.578716 17371 catalog_manager.cc:1347] Generated new cluster ID: 03bc3b95ceaa4ac590dec85800a04242
I20260810 23:31:05.578858 17371 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:31:05.595057 17371 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:31:05.595522 17371 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:31:05.614494 17371 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f95d40525cd84ef6b9c566db79bc6210: Generated new TSK 0
I20260810 23:31:05.615484 17371 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:31:05.623281 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 17215
W20260810 23:31:05.624540 17333 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.204:36093
W20260810 23:31:05.625365 15037 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 17215: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:31:05.625550 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 17215
2026-08-10T23:31:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2181 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:31:05 dist-test-slave-rmhg krb5kdc[17385](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:31:05 dist-test-slave-rmhg krb5kdc[17385](info): set up 2 sockets
Aug 10 23:31:05 dist-test-slave-rmhg krb5kdc[17385](info): commencing operation
krb5kdc: starting...
W20260810 23:31:06.872292 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.204s user 0.003s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:31:07 dist-test-slave-rmhg krb5kdc[17385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:31:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:31:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
I20260810 23:31:07.198923 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:44943
--webserver_interface=127.14.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:34515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:44943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:31:07.553107 17408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:31:07.553503 17408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:31:07.553637 17408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:31:07.561376 17408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:31:07.561501 17408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:31:07.561573 17408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:31:07.561653 17408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:31:07.561722 17408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:31:07.572240 17408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:34515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:44943
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:44943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:31:07.573764 17408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:31:07.575139 17408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:31:07.587954 17420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:07.587963 17427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:07.587929 17423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:31:07.588937 17408 server_base.cc:1034] running on GCE node
I20260810 23:31:07.589855 17408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:31:07.591439 17408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:31:07.592651 17408 hybrid_clock.cc:648] HybridClock initialized: now 1786404667592599 us; error 70 us; skew 500 ppm
I20260810 23:31:07.593034 17408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:31:07 dist-test-slave-rmhg krb5kdc[17385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404667, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:31:07.600445 17408 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:31:07.602275 17408 webserver.cc:460] Webserver started at http://127.14.175.126:42709/ using document root <none> and password file <none>
I20260810 23:31:07.602995 17408 fs_manager.cc:359] Metadata directory not provided
I20260810 23:31:07.603127 17408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:31:07.603435 17408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:31:07.606667 17408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "101b410327a344d7bcc1dbcef9e858cc"
format_stamp: "Formatted at 2026-08-10 23:31:07 on dist-test-slave-rmhg"
server_key: "ce96c67750c844a43fb0aba95a515441"
server_key_iv: "118f97cf593b75163fa689155af98fb5"
server_key_version: "encryptionkey@0"
I20260810 23:31:07.607492 17408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "101b410327a344d7bcc1dbcef9e858cc"
format_stamp: "Formatted at 2026-08-10 23:31:07 on dist-test-slave-rmhg"
server_key: "ce96c67750c844a43fb0aba95a515441"
server_key_iv: "118f97cf593b75163fa689155af98fb5"
server_key_version: "encryptionkey@0"
I20260810 23:31:07.613387 17408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 23:31:07.617309 17438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:07.618855 17408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:31:07.619071 17408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "101b410327a344d7bcc1dbcef9e858cc"
format_stamp: "Formatted at 2026-08-10 23:31:07 on dist-test-slave-rmhg"
server_key: "ce96c67750c844a43fb0aba95a515441"
server_key_iv: "118f97cf593b75163fa689155af98fb5"
server_key_version: "encryptionkey@0"
I20260810 23:31:07.619294 17408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:31:07.647742 17408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:31:07.652616 17408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:31:07.652930 17408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:31:07.672523 17408 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:44943
I20260810 23:31:07.672590 17526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:44943 every 8 connection(s)
I20260810 23:31:07.674060 17408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:31:07.678299 17527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:07.683997 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 17408
I20260810 23:31:07.684227 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:31:07.684813 15037 external_mini_cluster.cc:1428] Setting key e4bcec5d7ae26e8e159a8183707b7e6b
I20260810 23:31:07.690840 17527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 101b410327a344d7bcc1dbcef9e858cc: Bootstrap starting.
I20260810 23:31:07.694927 17527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 101b410327a344d7bcc1dbcef9e858cc: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:07.696254 17527 log.cc:824] T 00000000000000000000000000000000 P 101b410327a344d7bcc1dbcef9e858cc: Log is configured to *not* fsync() on all Append() calls
I20260810 23:31:07.699892 17527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 101b410327a344d7bcc1dbcef9e858cc: No bootstrap required, opened a new log
Aug 10 23:31:07 dist-test-slave-rmhg krb5kdc[17385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:31:07.710721 17527 raft_consensus.cc:348] T 00000000000000000000000000000000 P 101b410327a344d7bcc1dbcef9e858cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101b410327a344d7bcc1dbcef9e858cc" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 44943 } }
I20260810 23:31:07.711239 17527 raft_consensus.cc:374] T 00000000000000000000000000000000 P 101b410327a344d7bcc1dbcef9e858cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:07.711364 17530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:07.686918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52065 (local address 127.14.175.126:44943)
0810 23:31:07.687849 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:07.687862 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:07.687914 (+ 52us) server_negotiation.cc:407] Connection header received
0810 23:31:07.689800 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:07.689830 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:07.690293 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:07.690649 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:07.692576 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:07.694011 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:07.695764 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:07.696518 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:07.696649 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:07.701573 (+ 4924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:07.701587 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:07.701609 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:07.701678 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:07.705922 (+ 4244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:07.706956 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:07.706967 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:07.706978 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:07.707226 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:07.707751 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:07.707756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:07.707759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:07.708610 (+ 851us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:07.708948 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:07.709473 (+ 525us) server_negotiation.cc:299] Negotiation successful
0810 23:31:07.709882 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":427,"threads_started":1}
I20260810 23:31:07.711372 17527 raft_consensus.cc:729] T 00000000000000000000000000000000 P 101b410327a344d7bcc1dbcef9e858cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 101b410327a344d7bcc1dbcef9e858cc, State: Initialized, Role: FOLLOWER
I20260810 23:31:07.712026 17527 consensus_queue.cc:260] T 00000000000000000000000000000000 P 101b410327a344d7bcc1dbcef9e858cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101b410327a344d7bcc1dbcef9e858cc" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 44943 } }
I20260810 23:31:07.712286 17527 raft_consensus.cc:388] T 00000000000000000000000000000000 P 101b410327a344d7bcc1dbcef9e858cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:31:07.712424 17527 raft_consensus.cc:482] T 00000000000000000000000000000000 P 101b410327a344d7bcc1dbcef9e858cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:31:07.712604 17527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 101b410327a344d7bcc1dbcef9e858cc [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:07.715240 17527 raft_consensus.cc:504] T 00000000000000000000000000000000 P 101b410327a344d7bcc1dbcef9e858cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101b410327a344d7bcc1dbcef9e858cc" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 44943 } }
I20260810 23:31:07.715626 15037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:31:07.715837 17527 leader_election.cc:302] T 00000000000000000000000000000000 P 101b410327a344d7bcc1dbcef9e858cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 101b410327a344d7bcc1dbcef9e858cc; no voters:
I20260810 23:31:07.716528 17527 leader_election.cc:288] T 00000000000000000000000000000000 P 101b410327a344d7bcc1dbcef9e858cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:31:07.716631 17533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 101b410327a344d7bcc1dbcef9e858cc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:31:07.717847 17533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 101b410327a344d7bcc1dbcef9e858cc [term 1 LEADER]: Becoming Leader. State: Replica: 101b410327a344d7bcc1dbcef9e858cc, State: Running, Role: LEADER
I20260810 23:31:07.718420 17533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 101b410327a344d7bcc1dbcef9e858cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101b410327a344d7bcc1dbcef9e858cc" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 44943 } }
I20260810 23:31:07.718854 17527 sys_catalog.cc:564] T 00000000000000000000000000000000 P 101b410327a344d7bcc1dbcef9e858cc [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:31:07.721053 17535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 101b410327a344d7bcc1dbcef9e858cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 101b410327a344d7bcc1dbcef9e858cc. Latest consensus state: current_term: 1 leader_uuid: "101b410327a344d7bcc1dbcef9e858cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101b410327a344d7bcc1dbcef9e858cc" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 44943 } } }
I20260810 23:31:07.721455 17535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 101b410327a344d7bcc1dbcef9e858cc [sys.catalog]: This master's current role is: LEADER
I20260810 23:31:07.721935 17534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 101b410327a344d7bcc1dbcef9e858cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "101b410327a344d7bcc1dbcef9e858cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101b410327a344d7bcc1dbcef9e858cc" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 44943 } } }
I20260810 23:31:07.722316 17534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 101b410327a344d7bcc1dbcef9e858cc [sys.catalog]: This master's current role is: LEADER
I20260810 23:31:07.722376 17541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:31:07.738057 17541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:31:07.756132 17541 catalog_manager.cc:1347] Generated new cluster ID: a168b3e178c04cd58cf8c0aa800cd393
I20260810 23:31:07.756270 17541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:31:07.769716 17530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:07.736151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54105 (local address 127.14.175.126:44943)
0810 23:31:07.736347 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:07.736354 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:07.743422 (+ 7068us) server_negotiation.cc:407] Connection header received
0810 23:31:07.743962 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:07.743968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:07.744053 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:07.744140 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:07.746789 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:07.748192 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:07.750278 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:07.751004 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:07.751115 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:07.754870 (+ 3755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:07.754877 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:07.754883 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:07.754929 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:07.764634 (+ 9705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:07.766301 (+ 1667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:07.766310 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:07.766314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:07.766535 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:07.767244 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:07.767250 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:07.767254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:07.767813 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:07.768135 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:07.768979 (+ 844us) server_negotiation.cc:299] Negotiation successful
0810 23:31:07.769268 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 23:31:07.790488 17541 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:31:07.791908 17541 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:31:07.804023 17541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 101b410327a344d7bcc1dbcef9e858cc: Generated new TSK 0
I20260810 23:31:07.804929 17541 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:31:07.817251 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 17408
I20260810 23:31:07.818509 17520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:31:07.913035 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 17408
W20260810 23:31:07.937502 17383 connection.cc:535] client connection to 127.14.175.126:44943 recv error: Network error: failed to read from TLS socket (remote: 127.14.175.126:44943): Connection reset by peer (error 104)
2026-08-10T23:31:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2288 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:31:08 dist-test-slave-rmhg krb5kdc[17570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:31:08 dist-test-slave-rmhg krb5kdc[17570](info): set up 2 sockets
Aug 10 23:31:08 dist-test-slave-rmhg krb5kdc[17570](info): commencing operation
krb5kdc: starting...
W20260810 23:31:09.173481 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.217s user 0.000s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:31:09 dist-test-slave-rmhg krb5kdc[17570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:31:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:31:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
I20260810 23:31:09.510350 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:36973
--webserver_interface=127.14.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:35929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:36973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:31:09.865828 17590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:31:09.866149 17590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:31:09.866245 17590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:31:09.873653 17590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:31:09.873782 17590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:31:09.873831 17590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:31:09.873893 17590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:31:09.873939 17590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:31:09.884168 17590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:35929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:36973
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:36973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:31:09.885586 17590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:31:09.886924 17590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:31:09.898885 17596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:09.898909 17597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:09.898861 17602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:31:09.899698 17590 server_base.cc:1034] running on GCE node
I20260810 23:31:09.900555 17590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:31:09.902240 17590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:31:09.903470 17590 hybrid_clock.cc:648] HybridClock initialized: now 1786404669903405 us; error 64 us; skew 500 ppm
I20260810 23:31:09.903867 17590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:31:09 dist-test-slave-rmhg krb5kdc[17570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404669, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:31:09.907354 17590 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:31:09.908951 17590 webserver.cc:460] Webserver started at http://127.14.175.126:34259/ using document root <none> and password file <none>
I20260810 23:31:09.909729 17590 fs_manager.cc:359] Metadata directory not provided
I20260810 23:31:09.909854 17590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:31:09.910173 17590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:31:09.913550 17590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "a075cbb41de54f45ac5b0cc9a4c6a128"
format_stamp: "Formatted at 2026-08-10 23:31:09 on dist-test-slave-rmhg"
server_key: "771b7fba6c32a93212e20cebdfef4fea"
server_key_iv: "d8cf319cea1ce7e890ad63c3d581bf67"
server_key_version: "encryptionkey@0"
I20260810 23:31:09.914359 17590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "a075cbb41de54f45ac5b0cc9a4c6a128"
format_stamp: "Formatted at 2026-08-10 23:31:09 on dist-test-slave-rmhg"
server_key: "771b7fba6c32a93212e20cebdfef4fea"
server_key_iv: "d8cf319cea1ce7e890ad63c3d581bf67"
server_key_version: "encryptionkey@0"
I20260810 23:31:09.920003 17590 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:31:09.923929 17611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:09.925765 17590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 23:31:09.925997 17590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "a075cbb41de54f45ac5b0cc9a4c6a128"
format_stamp: "Formatted at 2026-08-10 23:31:09 on dist-test-slave-rmhg"
server_key: "771b7fba6c32a93212e20cebdfef4fea"
server_key_iv: "d8cf319cea1ce7e890ad63c3d581bf67"
server_key_version: "encryptionkey@0"
I20260810 23:31:09.926221 17590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:31:09.939733 17590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:31:09.944542 17590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:31:09.944847 17590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:31:09.963881 17590 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:36973
I20260810 23:31:09.963938 17693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:36973 every 8 connection(s)
I20260810 23:31:09.965493 17590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:31:09.969730 17694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:09.974303 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 17590
I20260810 23:31:09.974488 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:31:09.974931 15037 external_mini_cluster.cc:1428] Setting key 5d3155904618831838c826c1f5c565c0
I20260810 23:31:09.982379 17694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a075cbb41de54f45ac5b0cc9a4c6a128: Bootstrap starting.
I20260810 23:31:09.986382 17694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a075cbb41de54f45ac5b0cc9a4c6a128: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:09.987870 17694 log.cc:824] T 00000000000000000000000000000000 P a075cbb41de54f45ac5b0cc9a4c6a128: Log is configured to *not* fsync() on all Append() calls
I20260810 23:31:09.990495 17698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:09.976934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37739 (local address 127.14.175.126:36973)
0810 23:31:09.977882 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:09.977894 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:09.977943 (+ 49us) server_negotiation.cc:407] Connection header received
0810 23:31:09.980474 (+ 2531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:09.980504 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:09.980982 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:09.981430 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:09.982182 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:09.983612 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:09.985567 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:09.986312 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:09.986453 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:09.987861 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:09.987869 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:09.987886 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:31:09.988639 (+ 753us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:09.988743 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:31:09.988779 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:09.988969 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:09.989304 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 23:31:09.989486 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":483,"threads_started":1}
I20260810 23:31:09.991243 17694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a075cbb41de54f45ac5b0cc9a4c6a128: No bootstrap required, opened a new log
I20260810 23:31:09.995123 15037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:31:10.000247 17694 raft_consensus.cc:348] T 00000000000000000000000000000000 P a075cbb41de54f45ac5b0cc9a4c6a128 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a075cbb41de54f45ac5b0cc9a4c6a128" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 36973 } }
I20260810 23:31:10.000849 17694 raft_consensus.cc:374] T 00000000000000000000000000000000 P a075cbb41de54f45ac5b0cc9a4c6a128 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:10.000974 17694 raft_consensus.cc:729] T 00000000000000000000000000000000 P a075cbb41de54f45ac5b0cc9a4c6a128 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a075cbb41de54f45ac5b0cc9a4c6a128, State: Initialized, Role: FOLLOWER
I20260810 23:31:10.001791 17694 consensus_queue.cc:260] T 00000000000000000000000000000000 P a075cbb41de54f45ac5b0cc9a4c6a128 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a075cbb41de54f45ac5b0cc9a4c6a128" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 36973 } }
I20260810 23:31:10.002157 17694 raft_consensus.cc:388] T 00000000000000000000000000000000 P a075cbb41de54f45ac5b0cc9a4c6a128 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:31:10.002315 17694 raft_consensus.cc:482] T 00000000000000000000000000000000 P a075cbb41de54f45ac5b0cc9a4c6a128 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:31:10.002522 17694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a075cbb41de54f45ac5b0cc9a4c6a128 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:10.005570 17694 raft_consensus.cc:504] T 00000000000000000000000000000000 P a075cbb41de54f45ac5b0cc9a4c6a128 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a075cbb41de54f45ac5b0cc9a4c6a128" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 36973 } }
I20260810 23:31:10.006201 17694 leader_election.cc:302] T 00000000000000000000000000000000 P a075cbb41de54f45ac5b0cc9a4c6a128 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a075cbb41de54f45ac5b0cc9a4c6a128; no voters:
I20260810 23:31:10.007154 17694 leader_election.cc:288] T 00000000000000000000000000000000 P a075cbb41de54f45ac5b0cc9a4c6a128 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:31:10.007206 17698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:09.999691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36127 (local address 127.14.175.126:36973)
0810 23:31:09.999894 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:09.999901 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:10.000360 (+ 459us) server_negotiation.cc:407] Connection header received
0810 23:31:10.000542 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:10.000548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:10.000648 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:10.000770 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:10.001443 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:10.002611 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:10.004070 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:10.004855 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:10.004986 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:10.006271 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:10.006278 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:10.006283 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:31:10.006310 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:10.006365 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:31:10.006389 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:10.006517 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:10.006697 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 23:31:10.006769 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 23:31:10.007761 17706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a075cbb41de54f45ac5b0cc9a4c6a128 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:31:10.008535 17706 raft_consensus.cc:686] T 00000000000000000000000000000000 P a075cbb41de54f45ac5b0cc9a4c6a128 [term 1 LEADER]: Becoming Leader. State: Replica: a075cbb41de54f45ac5b0cc9a4c6a128, State: Running, Role: LEADER
I20260810 23:31:10.009392 17706 consensus_queue.cc:237] T 00000000000000000000000000000000 P a075cbb41de54f45ac5b0cc9a4c6a128 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a075cbb41de54f45ac5b0cc9a4c6a128" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 36973 } }
I20260810 23:31:10.009624 17694 sys_catalog.cc:564] T 00000000000000000000000000000000 P a075cbb41de54f45ac5b0cc9a4c6a128 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:31:10.012328 17708 sys_catalog.cc:455] T 00000000000000000000000000000000 P a075cbb41de54f45ac5b0cc9a4c6a128 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a075cbb41de54f45ac5b0cc9a4c6a128. Latest consensus state: current_term: 1 leader_uuid: "a075cbb41de54f45ac5b0cc9a4c6a128" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a075cbb41de54f45ac5b0cc9a4c6a128" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 36973 } } }
I20260810 23:31:10.012751 17708 sys_catalog.cc:458] T 00000000000000000000000000000000 P a075cbb41de54f45ac5b0cc9a4c6a128 [sys.catalog]: This master's current role is: LEADER
I20260810 23:31:10.012220 17707 sys_catalog.cc:455] T 00000000000000000000000000000000 P a075cbb41de54f45ac5b0cc9a4c6a128 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a075cbb41de54f45ac5b0cc9a4c6a128" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a075cbb41de54f45ac5b0cc9a4c6a128" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 36973 } } }
I20260810 23:31:10.013415 17707 sys_catalog.cc:458] T 00000000000000000000000000000000 P a075cbb41de54f45ac5b0cc9a4c6a128 [sys.catalog]: This master's current role is: LEADER
I20260810 23:31:10.018080 17720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:31:10.023948 17720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:31:10.035115 17720 catalog_manager.cc:1347] Generated new cluster ID: f16c144da2314336887a6d3adb850be9
I20260810 23:31:10.035261 17720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:31:10.053459 17720 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:31:10.054823 17720 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:31:10.063473 17720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a075cbb41de54f45ac5b0cc9a4c6a128: Generated new TSK 0
I20260810 23:31:10.064360 17720 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:31:10.075013 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 17590
W20260810 23:31:10.076751 17687 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37739
W20260810 23:31:10.077674 15037 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 17590: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:31:10.077860 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 17590
W20260810 23:31:10.102936 17567 connection.cc:535] client connection to 127.14.175.126:36973 recv error: Network error: failed to read from TLS socket (remote: 127.14.175.126:36973): Connection reset by peer (error 104)
2026-08-10T23:31:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2165 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:31:10 dist-test-slave-rmhg krb5kdc[17737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:31:10 dist-test-slave-rmhg krb5kdc[17737](info): set up 2 sockets
Aug 10 23:31:10 dist-test-slave-rmhg krb5kdc[17737](info): commencing operation
krb5kdc: starting...
W20260810 23:31:11.335722 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.213s user 0.001s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:31:11 dist-test-slave-rmhg krb5kdc[17737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:31:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:31:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
I20260810 23:31:11.670742 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:43341
--webserver_interface=127.14.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:40043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:43341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:31:12.020596 17762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:31:12.020932 17762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:31:12.021082 17762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:31:12.028623 17762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:31:12.028754 17762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:31:12.028832 17762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:31:12.028920 17762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:31:12.028977 17762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:31:12.039368 17762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:40043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:43341
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:43341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:31:12.040760 17762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:31:12.042207 17762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:31:12.054365 17773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:12.054380 17769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:12.054404 17768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:31:12.055265 17762 server_base.cc:1034] running on GCE node
I20260810 23:31:12.056090 17762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:31:12.057574 17762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:31:12.058794 17762 hybrid_clock.cc:648] HybridClock initialized: now 1786404672058729 us; error 53 us; skew 500 ppm
I20260810 23:31:12.059129 17762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:31:12 dist-test-slave-rmhg krb5kdc[17737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404672, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:31:12.062312 17762 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:31:12.063812 17762 webserver.cc:460] Webserver started at http://127.14.175.126:34487/ using document root <none> and password file <none>
I20260810 23:31:12.064536 17762 fs_manager.cc:359] Metadata directory not provided
I20260810 23:31:12.064661 17762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:31:12.064962 17762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:31:12.068114 17762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "fe89b2c25bf64760807e3b12ed0e4dec"
format_stamp: "Formatted at 2026-08-10 23:31:12 on dist-test-slave-rmhg"
server_key: "275228818b373475c8aa0cf641c20cd3"
server_key_iv: "fae180e85710cc0f11364eabe12213b0"
server_key_version: "encryptionkey@0"
I20260810 23:31:12.068869 17762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "fe89b2c25bf64760807e3b12ed0e4dec"
format_stamp: "Formatted at 2026-08-10 23:31:12 on dist-test-slave-rmhg"
server_key: "275228818b373475c8aa0cf641c20cd3"
server_key_iv: "fae180e85710cc0f11364eabe12213b0"
server_key_version: "encryptionkey@0"
I20260810 23:31:12.074131 17762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 23:31:12.077576 17785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:12.078950 17762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 23:31:12.079123 17762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "fe89b2c25bf64760807e3b12ed0e4dec"
format_stamp: "Formatted at 2026-08-10 23:31:12 on dist-test-slave-rmhg"
server_key: "275228818b373475c8aa0cf641c20cd3"
server_key_iv: "fae180e85710cc0f11364eabe12213b0"
server_key_version: "encryptionkey@0"
I20260810 23:31:12.079326 17762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:31:12.085312 17762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:31:12.089512 17762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:31:12.089792 17762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:31:12.108741 17762 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:43341
I20260810 23:31:12.108764 17868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:43341 every 8 connection(s)
I20260810 23:31:12.110247 17762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:31:12.113961 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 17762
I20260810 23:31:12.114130 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:31:12.114425 17869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:12.114605 15037 external_mini_cluster.cc:1428] Setting key 0d7802aba11d1e5fe28026dc6be826f9
I20260810 23:31:12.125227 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:12.116560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57799 (local address 127.14.175.126:43341)
0810 23:31:12.117589 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:12.117605 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:12.117961 (+ 356us) server_negotiation.cc:407] Connection header received
0810 23:31:12.120909 (+ 2948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:12.120947 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:12.121537 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:12.122091 (+ 554us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:12.123104 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:12.123114 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:12.123133 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:31:12.123151 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:12.123234 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:31:12.123263 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:12.123490 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:12.123624 (+ 134us) server_negotiation.cc:299] Negotiation successful
0810 23:31:12.123860 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":403,"threads_started":1}
I20260810 23:31:12.129016 17869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe89b2c25bf64760807e3b12ed0e4dec: Bootstrap starting.
I20260810 23:31:12.129251 15037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:31:12.132431 17869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe89b2c25bf64760807e3b12ed0e4dec: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:12.133683 17869 log.cc:824] T 00000000000000000000000000000000 P fe89b2c25bf64760807e3b12ed0e4dec: Log is configured to *not* fsync() on all Append() calls
I20260810 23:31:12.136373 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:12.133868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55067 (local address 127.14.175.126:43341)
0810 23:31:12.134056 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:12.134062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:12.134468 (+ 406us) server_negotiation.cc:407] Connection header received
0810 23:31:12.134677 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:12.134682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:12.134736 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:12.134830 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:12.135725 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:12.135729 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:12.135738 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:31:12.135754 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:12.135810 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:31:12.135826 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:12.135932 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:12.136039 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 23:31:12.136113 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:31:12.136843 17869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe89b2c25bf64760807e3b12ed0e4dec: No bootstrap required, opened a new log
I20260810 23:31:12.144830 17869 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe89b2c25bf64760807e3b12ed0e4dec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe89b2c25bf64760807e3b12ed0e4dec" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43341 } }
I20260810 23:31:12.145295 17869 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe89b2c25bf64760807e3b12ed0e4dec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:12.145401 17869 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe89b2c25bf64760807e3b12ed0e4dec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe89b2c25bf64760807e3b12ed0e4dec, State: Initialized, Role: FOLLOWER
I20260810 23:31:12.145970 17869 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe89b2c25bf64760807e3b12ed0e4dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe89b2c25bf64760807e3b12ed0e4dec" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43341 } }
I20260810 23:31:12.146198 17869 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe89b2c25bf64760807e3b12ed0e4dec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:31:12.146306 17869 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe89b2c25bf64760807e3b12ed0e4dec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:31:12.146438 17869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe89b2c25bf64760807e3b12ed0e4dec [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:12.148976 17869 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe89b2c25bf64760807e3b12ed0e4dec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe89b2c25bf64760807e3b12ed0e4dec" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43341 } }
I20260810 23:31:12.149540 17869 leader_election.cc:302] T 00000000000000000000000000000000 P fe89b2c25bf64760807e3b12ed0e4dec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe89b2c25bf64760807e3b12ed0e4dec; no voters:
I20260810 23:31:12.150125 17869 leader_election.cc:288] T 00000000000000000000000000000000 P fe89b2c25bf64760807e3b12ed0e4dec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:31:12.150260 17887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe89b2c25bf64760807e3b12ed0e4dec [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:31:12.150966 17887 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe89b2c25bf64760807e3b12ed0e4dec [term 1 LEADER]: Becoming Leader. State: Replica: fe89b2c25bf64760807e3b12ed0e4dec, State: Running, Role: LEADER
I20260810 23:31:12.151525 17887 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe89b2c25bf64760807e3b12ed0e4dec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe89b2c25bf64760807e3b12ed0e4dec" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43341 } }
I20260810 23:31:12.152303 17869 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe89b2c25bf64760807e3b12ed0e4dec [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:31:12.154368 17888 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe89b2c25bf64760807e3b12ed0e4dec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe89b2c25bf64760807e3b12ed0e4dec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe89b2c25bf64760807e3b12ed0e4dec" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43341 } } }
I20260810 23:31:12.154238 17892 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe89b2c25bf64760807e3b12ed0e4dec [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe89b2c25bf64760807e3b12ed0e4dec. Latest consensus state: current_term: 1 leader_uuid: "fe89b2c25bf64760807e3b12ed0e4dec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe89b2c25bf64760807e3b12ed0e4dec" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 43341 } } }
I20260810 23:31:12.154909 17888 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe89b2c25bf64760807e3b12ed0e4dec [sys.catalog]: This master's current role is: LEADER
I20260810 23:31:12.154909 17892 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe89b2c25bf64760807e3b12ed0e4dec [sys.catalog]: This master's current role is: LEADER
I20260810 23:31:12.155836 17904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:31:12.162204 17904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:31:12.172297 17904 catalog_manager.cc:1347] Generated new cluster ID: d9cd0e7b0d5d40dfb8c817be6e3767e0
I20260810 23:31:12.172428 17904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:31:12.189903 17904 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:31:12.190443 17904 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:31:12.210944 17904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe89b2c25bf64760807e3b12ed0e4dec: Generated new TSK 0
I20260810 23:31:12.212141 17904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:31:12.216426 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 17762
W20260810 23:31:12.217712 17864 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57799
W20260810 23:31:12.218519 15037 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 17762: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:31:12.218669 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 17762
2026-08-10T23:31:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2138 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:31:12 dist-test-slave-rmhg krb5kdc[17924](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:31:12 dist-test-slave-rmhg krb5kdc[17924](info): set up 2 sockets
Aug 10 23:31:12 dist-test-slave-rmhg krb5kdc[17924](info): commencing operation
krb5kdc: starting...
W20260810 23:31:13.484622 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.224s user 0.000s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:31:13 dist-test-slave-rmhg krb5kdc[17924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:31:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:31:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
I20260810 23:31:13.826278 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:37991
--webserver_interface=127.14.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:40503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:37991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:31:14.175985 17948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:31:14.176316 17948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:31:14.176445 17948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:31:14.184275 17948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:31:14.184420 17948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:31:14.184506 17948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:31:14.184607 17948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:31:14.184662 17948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:31:14.195072 17948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:40503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:37991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:37991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:31:14.196502 17948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:31:14.197932 17948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:31:14.210233 17954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:14.210306 17955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:14.210278 17958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:31:14.210898 17948 server_base.cc:1034] running on GCE node
I20260810 23:31:14.211748 17948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:31:14.213325 17948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:31:14.214534 17948 hybrid_clock.cc:648] HybridClock initialized: now 1786404674214483 us; error 47 us; skew 500 ppm
I20260810 23:31:14.214900 17948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:31:14 dist-test-slave-rmhg krb5kdc[17924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404674, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:31:14.223572 17948 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:31:14.225571 17948 webserver.cc:460] Webserver started at http://127.14.175.126:44695/ using document root <none> and password file <none>
I20260810 23:31:14.226408 17948 fs_manager.cc:359] Metadata directory not provided
I20260810 23:31:14.226557 17948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:31:14.226897 17948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:31:14.230378 17948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "dff834df13854cf288e356c9ed38cb4a"
format_stamp: "Formatted at 2026-08-10 23:31:14 on dist-test-slave-rmhg"
server_key: "924411c5874196432d94472fe92336e9"
server_key_iv: "0bfc32acebdb07b9554478a74614b41b"
server_key_version: "encryptionkey@0"
I20260810 23:31:14.231251 17948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "dff834df13854cf288e356c9ed38cb4a"
format_stamp: "Formatted at 2026-08-10 23:31:14 on dist-test-slave-rmhg"
server_key: "924411c5874196432d94472fe92336e9"
server_key_iv: "0bfc32acebdb07b9554478a74614b41b"
server_key_version: "encryptionkey@0"
I20260810 23:31:14.237250 17948 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260810 23:31:14.241405 17965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:14.243005 17948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 23:31:14.243207 17948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "dff834df13854cf288e356c9ed38cb4a"
format_stamp: "Formatted at 2026-08-10 23:31:14 on dist-test-slave-rmhg"
server_key: "924411c5874196432d94472fe92336e9"
server_key_iv: "0bfc32acebdb07b9554478a74614b41b"
server_key_version: "encryptionkey@0"
I20260810 23:31:14.243420 17948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:31:14.274825 17948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:31:14.279232 17948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:31:14.279531 17948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:31:14.297947 17948 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:37991
I20260810 23:31:14.297992 18049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:37991 every 8 connection(s)
I20260810 23:31:14.299364 17948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:31:14.303428 18050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:14.308615 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 17948
I20260810 23:31:14.308784 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:31:14.309264 15037 external_mini_cluster.cc:1428] Setting key b86e3befad6bbc6907be6d05c3091cc3
I20260810 23:31:14.317034 18050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff834df13854cf288e356c9ed38cb4a: Bootstrap starting.
I20260810 23:31:14.320987 18050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dff834df13854cf288e356c9ed38cb4a: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:14.322477 18050 log.cc:824] T 00000000000000000000000000000000 P dff834df13854cf288e356c9ed38cb4a: Log is configured to *not* fsync() on all Append() calls
I20260810 23:31:14.325922 18050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff834df13854cf288e356c9ed38cb4a: No bootstrap required, opened a new log
Aug 10 23:31:14 dist-test-slave-rmhg krb5kdc[17924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:31:14.336542 18050 raft_consensus.cc:348] T 00000000000000000000000000000000 P dff834df13854cf288e356c9ed38cb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff834df13854cf288e356c9ed38cb4a" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37991 } }
I20260810 23:31:14.337061 18050 raft_consensus.cc:374] T 00000000000000000000000000000000 P dff834df13854cf288e356c9ed38cb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:14.337226 18054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:14.311244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55481 (local address 127.14.175.126:37991)
0810 23:31:14.312116 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:14.312141 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:14.312201 (+ 60us) server_negotiation.cc:407] Connection header received
0810 23:31:14.315049 (+ 2848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:14.315086 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:14.315685 (+ 599us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:14.316157 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:14.317739 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:14.319833 (+ 2094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:14.321424 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:14.322572 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:14.322778 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:14.327180 (+ 4402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:14.327192 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:14.327214 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:14.327270 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:14.331558 (+ 4288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:14.332478 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:14.332489 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:14.332499 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:14.332773 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:14.333333 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:14.333338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:14.333342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:14.334232 (+ 890us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:14.334568 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:14.335117 (+ 549us) server_negotiation.cc:299] Negotiation successful
0810 23:31:14.335676 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":381,"threads_started":1}
I20260810 23:31:14.337234 18050 raft_consensus.cc:729] T 00000000000000000000000000000000 P dff834df13854cf288e356c9ed38cb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff834df13854cf288e356c9ed38cb4a, State: Initialized, Role: FOLLOWER
I20260810 23:31:14.337934 18050 consensus_queue.cc:260] T 00000000000000000000000000000000 P dff834df13854cf288e356c9ed38cb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff834df13854cf288e356c9ed38cb4a" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37991 } }
I20260810 23:31:14.338220 18050 raft_consensus.cc:388] T 00000000000000000000000000000000 P dff834df13854cf288e356c9ed38cb4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:31:14.338361 18050 raft_consensus.cc:482] T 00000000000000000000000000000000 P dff834df13854cf288e356c9ed38cb4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:31:14.338558 18050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dff834df13854cf288e356c9ed38cb4a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:14.341794 18050 raft_consensus.cc:504] T 00000000000000000000000000000000 P dff834df13854cf288e356c9ed38cb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff834df13854cf288e356c9ed38cb4a" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37991 } }
I20260810 23:31:14.342161 15037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:31:14.342414 18050 leader_election.cc:302] T 00000000000000000000000000000000 P dff834df13854cf288e356c9ed38cb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dff834df13854cf288e356c9ed38cb4a; no voters:
I20260810 23:31:14.343029 18050 leader_election.cc:288] T 00000000000000000000000000000000 P dff834df13854cf288e356c9ed38cb4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:31:14.343207 18057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dff834df13854cf288e356c9ed38cb4a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:31:14.345268 18050 sys_catalog.cc:564] T 00000000000000000000000000000000 P dff834df13854cf288e356c9ed38cb4a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:31:14.356453 18057 raft_consensus.cc:686] T 00000000000000000000000000000000 P dff834df13854cf288e356c9ed38cb4a [term 1 LEADER]: Becoming Leader. State: Replica: dff834df13854cf288e356c9ed38cb4a, State: Running, Role: LEADER
I20260810 23:31:14.357249 18057 consensus_queue.cc:237] T 00000000000000000000000000000000 P dff834df13854cf288e356c9ed38cb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff834df13854cf288e356c9ed38cb4a" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37991 } }
I20260810 23:31:14.360335 18058 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff834df13854cf288e356c9ed38cb4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dff834df13854cf288e356c9ed38cb4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff834df13854cf288e356c9ed38cb4a" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37991 } } }
I20260810 23:31:14.360725 18058 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff834df13854cf288e356c9ed38cb4a [sys.catalog]: This master's current role is: LEADER
I20260810 23:31:14.361864 18060 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff834df13854cf288e356c9ed38cb4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader dff834df13854cf288e356c9ed38cb4a. Latest consensus state: current_term: 1 leader_uuid: "dff834df13854cf288e356c9ed38cb4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff834df13854cf288e356c9ed38cb4a" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37991 } } }
I20260810 23:31:14.362185 18091 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 23:31:14.362206 18060 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff834df13854cf288e356c9ed38cb4a [sys.catalog]: This master's current role is: LEADER
I20260810 23:31:14.363255 18088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:31:14.368435 18088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:31:14.374833 18054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:14.358117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53215 (local address 127.14.175.126:37991)
0810 23:31:14.358320 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:14.358328 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:14.358772 (+ 444us) server_negotiation.cc:407] Connection header received
0810 23:31:14.359033 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:14.359039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:14.359158 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:14.359263 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:14.361314 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:14.362444 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:14.364300 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:14.364788 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:14.364892 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:14.368030 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:14.368037 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:14.368042 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:14.368088 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:14.371952 (+ 3864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:14.372745 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:14.372750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:14.372753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:14.372894 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:14.373319 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:14.373323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:14.373325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:14.373655 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:14.373878 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:14.374252 (+ 374us) server_negotiation.cc:299] Negotiation successful
0810 23:31:14.374471 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 23:31:14.379217 18088 catalog_manager.cc:1347] Generated new cluster ID: fcc63982d42a4b308a0bdb4175d950ee
I20260810 23:31:14.379410 18088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:31:14.394912 18088 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:31:14.396273 18088 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:31:14.408667 18088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dff834df13854cf288e356c9ed38cb4a: Generated new TSK 0
I20260810 23:31:14.409896 18088 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:31:14.424933 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 17948
I20260810 23:31:14.426255 18044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:31:14.520627 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 17948
2026-08-10T23:31:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:31:14 dist-test-slave-rmhg krb5kdc[18103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:31:14 dist-test-slave-rmhg krb5kdc[18103](info): set up 2 sockets
Aug 10 23:31:14 dist-test-slave-rmhg krb5kdc[18103](info): commencing operation
krb5kdc: starting...
W20260810 23:31:15.792022 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.222s user 0.000s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:31:16 dist-test-slave-rmhg krb5kdc[18103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:31:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:31:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
I20260810 23:31:16.128062 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:42253
--webserver_interface=127.14.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:34211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:42253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:31:16.489355 18120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:31:16.489668 18120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:31:16.489790 18120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:31:16.497087 18120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:31:16.497239 18120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:31:16.497316 18120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:31:16.497406 18120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:31:16.497465 18120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:31:16.507504 18120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:34211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:42253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:42253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:31:16.508839 18120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:31:16.510218 18120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:31:16.522105 18133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:16.522209 18128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:16.522205 18129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:31:16.523505 18120 server_base.cc:1034] running on GCE node
I20260810 23:31:16.524369 18120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:31:16.525914 18120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:31:16.527169 18120 hybrid_clock.cc:648] HybridClock initialized: now 1786404676527110 us; error 46 us; skew 500 ppm
I20260810 23:31:16.527556 18120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:31:16 dist-test-slave-rmhg krb5kdc[18103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404676, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:31:16.530810 18120 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:31:16.532305 18120 webserver.cc:460] Webserver started at http://127.14.175.126:33167/ using document root <none> and password file <none>
I20260810 23:31:16.532980 18120 fs_manager.cc:359] Metadata directory not provided
I20260810 23:31:16.533087 18120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:31:16.533418 18120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:31:16.536473 18120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "94f1c8302e3e4a7eaeb65d91cb505596"
format_stamp: "Formatted at 2026-08-10 23:31:16 on dist-test-slave-rmhg"
server_key: "bd23a67a4b979e726efbb843f44e6456"
server_key_iv: "66ecbce46f5b4b0edb2eaab32a67bcff"
server_key_version: "encryptionkey@0"
I20260810 23:31:16.537241 18120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "94f1c8302e3e4a7eaeb65d91cb505596"
format_stamp: "Formatted at 2026-08-10 23:31:16 on dist-test-slave-rmhg"
server_key: "bd23a67a4b979e726efbb843f44e6456"
server_key_iv: "66ecbce46f5b4b0edb2eaab32a67bcff"
server_key_version: "encryptionkey@0"
I20260810 23:31:16.542562 18120 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 23:31:16.546330 18144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:16.547827 18120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:31:16.548019 18120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "94f1c8302e3e4a7eaeb65d91cb505596"
format_stamp: "Formatted at 2026-08-10 23:31:16 on dist-test-slave-rmhg"
server_key: "bd23a67a4b979e726efbb843f44e6456"
server_key_iv: "66ecbce46f5b4b0edb2eaab32a67bcff"
server_key_version: "encryptionkey@0"
I20260810 23:31:16.548237 18120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:31:16.581202 18120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:31:16.585886 18120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:31:16.586193 18120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:31:16.604995 18120 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:42253
I20260810 23:31:16.605019 18220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:42253 every 8 connection(s)
I20260810 23:31:16.606504 18120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:31:16.610549 18221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:16.612566 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 18120
I20260810 23:31:16.612774 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:31:16.613427 15037 external_mini_cluster.cc:1428] Setting key 97098c5061bdb45844d19269de644e7c
I20260810 23:31:16.623492 18221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94f1c8302e3e4a7eaeb65d91cb505596: Bootstrap starting.
I20260810 23:31:16.627419 18221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94f1c8302e3e4a7eaeb65d91cb505596: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:16.627955 18224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:16.615359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36017 (local address 127.14.175.126:42253)
0810 23:31:16.616274 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:16.616287 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:16.616334 (+ 47us) server_negotiation.cc:407] Connection header received
0810 23:31:16.618134 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:16.618165 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:16.618593 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:16.618934 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:16.619617 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:16.621100 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:16.623043 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:16.623940 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:16.624106 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:16.625459 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:16.625468 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:16.625487 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:31:16.626091 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:16.626180 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:31:16.626209 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:16.626306 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:16.626532 (+ 226us) server_negotiation.cc:299] Negotiation successful
0810 23:31:16.626819 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":400,"threads_started":1}
I20260810 23:31:16.628647 18221 log.cc:824] T 00000000000000000000000000000000 P 94f1c8302e3e4a7eaeb65d91cb505596: Log is configured to *not* fsync() on all Append() calls
I20260810 23:31:16.631968 18221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94f1c8302e3e4a7eaeb65d91cb505596: No bootstrap required, opened a new log
I20260810 23:31:16.632123 15037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:31:16.642535 18221 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94f1c8302e3e4a7eaeb65d91cb505596 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f1c8302e3e4a7eaeb65d91cb505596" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 42253 } }
I20260810 23:31:16.643153 18221 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94f1c8302e3e4a7eaeb65d91cb505596 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:16.643277 18221 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94f1c8302e3e4a7eaeb65d91cb505596 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94f1c8302e3e4a7eaeb65d91cb505596, State: Initialized, Role: FOLLOWER
I20260810 23:31:16.644025 18221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94f1c8302e3e4a7eaeb65d91cb505596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f1c8302e3e4a7eaeb65d91cb505596" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 42253 } }
I20260810 23:31:16.644361 18221 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94f1c8302e3e4a7eaeb65d91cb505596 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:31:16.644584 18221 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94f1c8302e3e4a7eaeb65d91cb505596 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:31:16.644685 18224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:16.637636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50991 (local address 127.14.175.126:42253)
0810 23:31:16.637852 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:16.637858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:16.638196 (+ 338us) server_negotiation.cc:407] Connection header received
0810 23:31:16.638393 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:16.638397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:16.638466 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:16.638595 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:16.639141 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:16.639925 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:16.641985 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:16.642453 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:16.642567 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:16.643834 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:16.643840 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:16.643843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:31:16.643863 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:16.643905 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:31:16.643922 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:16.643995 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:16.644166 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 23:31:16.644255 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 23:31:16.644799 18221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94f1c8302e3e4a7eaeb65d91cb505596 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:16.647714 18221 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94f1c8302e3e4a7eaeb65d91cb505596 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f1c8302e3e4a7eaeb65d91cb505596" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 42253 } }
I20260810 23:31:16.648301 18221 leader_election.cc:302] T 00000000000000000000000000000000 P 94f1c8302e3e4a7eaeb65d91cb505596 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94f1c8302e3e4a7eaeb65d91cb505596; no voters:
I20260810 23:31:16.648959 18221 leader_election.cc:288] T 00000000000000000000000000000000 P 94f1c8302e3e4a7eaeb65d91cb505596 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:31:16.649106 18233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94f1c8302e3e4a7eaeb65d91cb505596 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:31:16.649919 18233 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94f1c8302e3e4a7eaeb65d91cb505596 [term 1 LEADER]: Becoming Leader. State: Replica: 94f1c8302e3e4a7eaeb65d91cb505596, State: Running, Role: LEADER
I20260810 23:31:16.650527 18233 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94f1c8302e3e4a7eaeb65d91cb505596 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f1c8302e3e4a7eaeb65d91cb505596" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 42253 } }
I20260810 23:31:16.651248 18221 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94f1c8302e3e4a7eaeb65d91cb505596 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:31:16.653864 18235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94f1c8302e3e4a7eaeb65d91cb505596 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94f1c8302e3e4a7eaeb65d91cb505596. Latest consensus state: current_term: 1 leader_uuid: "94f1c8302e3e4a7eaeb65d91cb505596" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f1c8302e3e4a7eaeb65d91cb505596" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 42253 } } }
I20260810 23:31:16.653842 18234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94f1c8302e3e4a7eaeb65d91cb505596 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94f1c8302e3e4a7eaeb65d91cb505596" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f1c8302e3e4a7eaeb65d91cb505596" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 42253 } } }
I20260810 23:31:16.654390 18234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94f1c8302e3e4a7eaeb65d91cb505596 [sys.catalog]: This master's current role is: LEADER
I20260810 23:31:16.654393 18235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94f1c8302e3e4a7eaeb65d91cb505596 [sys.catalog]: This master's current role is: LEADER
I20260810 23:31:16.655314 18242 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:31:16.667891 18242 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:31:16.678545 18242 catalog_manager.cc:1347] Generated new cluster ID: 69d26f933762407f99b264823cc839c0
I20260810 23:31:16.678694 18242 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:31:16.699244 18242 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:31:16.700681 18242 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:31:16.708287 18242 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94f1c8302e3e4a7eaeb65d91cb505596: Generated new TSK 0
I20260810 23:31:16.709276 18242 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:31:16.716455 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 18120
W20260810 23:31:16.717747 18215 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36017
W20260810 23:31:16.718482 15037 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 18120: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:31:16.718639 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 18120
2026-08-10T23:31:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2192 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:31:16 dist-test-slave-rmhg krb5kdc[18266](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:31:16 dist-test-slave-rmhg krb5kdc[18266](info): set up 2 sockets
Aug 10 23:31:16 dist-test-slave-rmhg krb5kdc[18266](info): commencing operation
krb5kdc: starting...
W20260810 23:31:17.971630 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.209s user 0.001s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:31:18 dist-test-slave-rmhg krb5kdc[18266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:31:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:31:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
I20260810 23:31:18.317806 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:35399
--webserver_interface=127.14.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:32923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:35399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:31:18.686978 18286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:31:18.687311 18286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:31:18.687459 18286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:31:18.694902 18286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:31:18.695052 18286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:31:18.695134 18286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:31:18.695216 18286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:31:18.695277 18286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:31:18.705915 18286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:32923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:35399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:35399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:31:18.707360 18286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:31:18.708739 18286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:31:18.721094 18292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:18.721115 18293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:18.721170 18295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:31:18.722115 18286 server_base.cc:1034] running on GCE node
I20260810 23:31:18.722981 18286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:31:18.724483 18286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:31:18.725733 18286 hybrid_clock.cc:648] HybridClock initialized: now 1786404678725671 us; error 57 us; skew 500 ppm
I20260810 23:31:18.726094 18286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:31:18 dist-test-slave-rmhg krb5kdc[18266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404678, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:31:18.729339 18286 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:31:18.731199 18286 webserver.cc:460] Webserver started at http://127.14.175.126:43981/ using document root <none> and password file <none>
I20260810 23:31:18.732000 18286 fs_manager.cc:359] Metadata directory not provided
I20260810 23:31:18.732144 18286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:31:18.732460 18286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:31:18.735743 18286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "656a3097b06948848fbdf80df17a3169"
format_stamp: "Formatted at 2026-08-10 23:31:18 on dist-test-slave-rmhg"
server_key: "2d8c750597aa487403545592e9c546a9"
server_key_iv: "01f104795a7e9343f63d6828879bdd1a"
server_key_version: "encryptionkey@0"
I20260810 23:31:18.736582 18286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "656a3097b06948848fbdf80df17a3169"
format_stamp: "Formatted at 2026-08-10 23:31:18 on dist-test-slave-rmhg"
server_key: "2d8c750597aa487403545592e9c546a9"
server_key_iv: "01f104795a7e9343f63d6828879bdd1a"
server_key_version: "encryptionkey@0"
I20260810 23:31:18.742381 18286 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 23:31:18.746523 18306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:18.748077 18286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260810 23:31:18.748289 18286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "656a3097b06948848fbdf80df17a3169"
format_stamp: "Formatted at 2026-08-10 23:31:18 on dist-test-slave-rmhg"
server_key: "2d8c750597aa487403545592e9c546a9"
server_key_iv: "01f104795a7e9343f63d6828879bdd1a"
server_key_version: "encryptionkey@0"
I20260810 23:31:18.748504 18286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:31:18.754755 18286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:31:18.758764 18286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:31:18.759049 18286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:31:18.777849 18286 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:35399
I20260810 23:31:18.777884 18378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:35399 every 8 connection(s)
I20260810 23:31:18.779359 18286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:31:18.783499 18382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:18.784518 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 18286
I20260810 23:31:18.784726 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:31:18.785315 15037 external_mini_cluster.cc:1428] Setting key 07a65f2fbd80625e297e7fb8c3ef6c83
I20260810 23:31:18.793841 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:18.787279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38731 (local address 127.14.175.126:35399)
0810 23:31:18.788097 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:18.788126 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:18.788173 (+ 47us) server_negotiation.cc:407] Connection header received
0810 23:31:18.790199 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:18.790228 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:18.790621 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:18.791005 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:18.792097 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:18.792110 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:18.792126 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:31:18.792142 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:18.792214 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:31:18.792236 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:18.792449 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:18.792630 (+ 181us) server_negotiation.cc:299] Negotiation successful
0810 23:31:18.792819 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":356,"threads_started":1}
I20260810 23:31:18.797197 18382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 656a3097b06948848fbdf80df17a3169: Bootstrap starting.
I20260810 23:31:18.797570 15037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:31:18.800431 18382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 656a3097b06948848fbdf80df17a3169: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:18.801770 18382 log.cc:824] T 00000000000000000000000000000000 P 656a3097b06948848fbdf80df17a3169: Log is configured to *not* fsync() on all Append() calls
I20260810 23:31:18.804445 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:18.801837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37663 (local address 127.14.175.126:35399)
0810 23:31:18.802050 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:18.802056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:18.802514 (+ 458us) server_negotiation.cc:407] Connection header received
0810 23:31:18.802715 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:18.802721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:18.802780 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:18.802903 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:18.803842 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:18.803847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:18.803850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:31:18.803866 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:18.803906 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:31:18.803921 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:18.804007 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:18.804123 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 23:31:18.804172 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 23:31:18.805402 18382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 656a3097b06948848fbdf80df17a3169: No bootstrap required, opened a new log
I20260810 23:31:18.813390 18382 raft_consensus.cc:348] T 00000000000000000000000000000000 P 656a3097b06948848fbdf80df17a3169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656a3097b06948848fbdf80df17a3169" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 35399 } }
I20260810 23:31:18.813819 18382 raft_consensus.cc:374] T 00000000000000000000000000000000 P 656a3097b06948848fbdf80df17a3169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:18.813916 18382 raft_consensus.cc:729] T 00000000000000000000000000000000 P 656a3097b06948848fbdf80df17a3169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 656a3097b06948848fbdf80df17a3169, State: Initialized, Role: FOLLOWER
I20260810 23:31:18.814539 18382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 656a3097b06948848fbdf80df17a3169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656a3097b06948848fbdf80df17a3169" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 35399 } }
I20260810 23:31:18.814788 18382 raft_consensus.cc:388] T 00000000000000000000000000000000 P 656a3097b06948848fbdf80df17a3169 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:31:18.814926 18382 raft_consensus.cc:482] T 00000000000000000000000000000000 P 656a3097b06948848fbdf80df17a3169 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:31:18.815081 18382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 656a3097b06948848fbdf80df17a3169 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:18.817376 18382 raft_consensus.cc:504] T 00000000000000000000000000000000 P 656a3097b06948848fbdf80df17a3169 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656a3097b06948848fbdf80df17a3169" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 35399 } }
I20260810 23:31:18.817890 18382 leader_election.cc:302] T 00000000000000000000000000000000 P 656a3097b06948848fbdf80df17a3169 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 656a3097b06948848fbdf80df17a3169; no voters:
I20260810 23:31:18.818514 18382 leader_election.cc:288] T 00000000000000000000000000000000 P 656a3097b06948848fbdf80df17a3169 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:31:18.818655 18400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 656a3097b06948848fbdf80df17a3169 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:31:18.819329 18400 raft_consensus.cc:686] T 00000000000000000000000000000000 P 656a3097b06948848fbdf80df17a3169 [term 1 LEADER]: Becoming Leader. State: Replica: 656a3097b06948848fbdf80df17a3169, State: Running, Role: LEADER
I20260810 23:31:18.819947 18400 consensus_queue.cc:237] T 00000000000000000000000000000000 P 656a3097b06948848fbdf80df17a3169 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656a3097b06948848fbdf80df17a3169" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 35399 } }
I20260810 23:31:18.820784 18382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 656a3097b06948848fbdf80df17a3169 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:31:18.822855 18402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 656a3097b06948848fbdf80df17a3169 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "656a3097b06948848fbdf80df17a3169" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656a3097b06948848fbdf80df17a3169" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 35399 } } }
I20260810 23:31:18.823071 18404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 656a3097b06948848fbdf80df17a3169 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 656a3097b06948848fbdf80df17a3169. Latest consensus state: current_term: 1 leader_uuid: "656a3097b06948848fbdf80df17a3169" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656a3097b06948848fbdf80df17a3169" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 35399 } } }
I20260810 23:31:18.823328 18402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 656a3097b06948848fbdf80df17a3169 [sys.catalog]: This master's current role is: LEADER
I20260810 23:31:18.823386 18404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 656a3097b06948848fbdf80df17a3169 [sys.catalog]: This master's current role is: LEADER
I20260810 23:31:18.825379 18418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:31:18.829978 18418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:31:18.838589 18418 catalog_manager.cc:1347] Generated new cluster ID: 05a63caeb0ef43b6921dfd2922c62e72
I20260810 23:31:18.838728 18418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:31:18.852983 18418 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:31:18.853654 18418 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:31:18.864850 18418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 656a3097b06948848fbdf80df17a3169: Generated new TSK 0
I20260810 23:31:18.866137 18418 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:31:18.882949 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 18286
W20260810 23:31:18.884851 18374 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38731
W20260810 23:31:18.885708 15037 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 18286: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:31:18.885874 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 18286
2026-08-10T23:31:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2171 ms)
[----------] 18 tests from AuthTokenIssuingTest (39860 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:31:19 dist-test-slave-rmhg krb5kdc[18437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:31:19 dist-test-slave-rmhg krb5kdc[18437](info): set up 2 sockets
Aug 10 23:31:19 dist-test-slave-rmhg krb5kdc[18437](info): commencing operation
krb5kdc: starting...
W20260810 23:31:20.143255 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.207s user 0.003s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:31:20 dist-test-slave-rmhg krb5kdc[18437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:31:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:31:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
I20260810 23:31:20.516817 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:37997
--webserver_interface=127.14.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:37981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:37997,127.14.175.125:35865,127.14.175.124:42283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:31:20.879405 18459 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:31:20.879796 18459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:31:20.879871 18459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:31:20.879967 18459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:31:20.887635 18459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:31:20.887760 18459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:31:20.887835 18459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:31:20.887914 18459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:31:20.888007 18459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:31:20.898608 18459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:37981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:37997,127.14.175.125:35865,127.14.175.124:42283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:37997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:31:20.900059 18459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:31:20.901542 18459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:31:20.914142 18469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:20.914114 18470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:20.914197 18473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:31:20.914929 18459 server_base.cc:1034] running on GCE node
I20260810 23:31:20.915817 18459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:31:20.917411 18459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:31:20.918658 18459 hybrid_clock.cc:648] HybridClock initialized: now 1786404680918585 us; error 45 us; skew 500 ppm
I20260810 23:31:20.919050 18459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:31:20 dist-test-slave-rmhg krb5kdc[18437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404680, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:31:20.927143 18459 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:31:20.928844 18459 webserver.cc:460] Webserver started at http://127.14.175.126:36917/ using document root <none> and password file <none>
I20260810 23:31:20.929692 18459 fs_manager.cc:359] Metadata directory not provided
I20260810 23:31:20.929802 18459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:31:20.930146 18459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:31:20.933441 18459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "31f9bdd654f047bb8c98a5e05e8bff7c"
format_stamp: "Formatted at 2026-08-10 23:31:20 on dist-test-slave-rmhg"
server_key: "a7644881b01adbf4730adcf861689a9f"
server_key_iv: "da03b6f1803853715a1178aafd8b02f4"
server_key_version: "encryptionkey@0"
I20260810 23:31:20.934274 18459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "31f9bdd654f047bb8c98a5e05e8bff7c"
format_stamp: "Formatted at 2026-08-10 23:31:20 on dist-test-slave-rmhg"
server_key: "a7644881b01adbf4730adcf861689a9f"
server_key_iv: "da03b6f1803853715a1178aafd8b02f4"
server_key_version: "encryptionkey@0"
I20260810 23:31:20.939787 18459 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260810 23:31:20.943468 18493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:20.945016 18459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:31:20.945236 18459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "31f9bdd654f047bb8c98a5e05e8bff7c"
format_stamp: "Formatted at 2026-08-10 23:31:20 on dist-test-slave-rmhg"
server_key: "a7644881b01adbf4730adcf861689a9f"
server_key_iv: "da03b6f1803853715a1178aafd8b02f4"
server_key_version: "encryptionkey@0"
I20260810 23:31:20.945468 18459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:31:20.961925 18459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:31:20.966455 18459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:31:20.966746 18459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:31:20.986027 18459 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:37997
I20260810 23:31:20.986042 18570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:37997 every 8 connection(s)
I20260810 23:31:20.987648 18459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:31:20.991737 18574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:20.992477 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 18459
I20260810 23:31:20.992690 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:31:20.993412 15037 external_mini_cluster.cc:1428] Setting key 8d4e62ab9a30f1de5920f6d24b42b0b5
I20260810 23:31:20.998955 18574 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37997 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:31:21 dist-test-slave-rmhg krb5kdc[18437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404680, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:31:21.024351 18578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.002637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55082 (local address 127.14.175.126:37997)
0810 23:31:21.004742 (+ 2105us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:21.004752 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:21.004803 (+ 51us) server_negotiation.cc:407] Connection header received
0810 23:31:21.006849 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:21.006857 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:21.007135 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:21.007310 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:21.008780 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:21.010222 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:21.011659 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:21.012552 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:21.012752 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:21.016547 (+ 3795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:21.016556 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:21.016572 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:21.016610 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:21.019526 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:21.020469 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:21.020478 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:21.020482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:21.020675 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:21.021227 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:21.021231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:21.021233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:21.021837 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:21.022032 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:21.022746 (+ 714us) server_negotiation.cc:299] Negotiation successful
0810 23:31:21.023057 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1890,"spinlock_wait_cycles":29568,"thread_start_us":1539,"threads_started":1}
I20260810 23:31:21.024389 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.002637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:37997 (local address 127.0.0.1:55082)
0810 23:31:21.003636 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:21.003717 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:21.006374 (+ 2657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:21.007395 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:21.007403 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:21.008144 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:21.008532 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:21.008551 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:21.010379 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:21.010383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:21.011504 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:21.011517 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:21.012785 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:21.012789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:21.012925 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:21.013597 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:21.013622 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:21.016260 (+ 2638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:21.019687 (+ 3427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:21.019701 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:21.019715 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:21.020304 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:21.020805 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:21.020809 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:21.020812 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:21.021050 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:21.022001 (+ 951us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:21.022011 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:21.022332 (+ 321us) client_negotiation.cc:769] Sending connection context
0810 23:31:21.022630 (+ 298us) client_negotiation.cc:240] Negotiation successful
0810 23:31:21.023245 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":68,"thread_start_us":397,"threads_started":1}
I20260810 23:31:21.027838 18574 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.125" port: 35865 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.175.125@KRBTEST.COM; defaulting to no policy
I20260810 23:31:21.028861 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.028480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:35865 (local address 127.0.0.1:40082)
0810 23:31:21.028624 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:21.028740 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.125:35865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 23:31:21.029095 18500 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.175.125:35865: connect: Connection refused (error 111)
I20260810 23:31:21.030854 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.030422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:35865 (local address 127.0.0.1:40092)
0810 23:31:21.030630 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:21.030709 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.125:35865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.125@KRBTEST.COM" created.
W20260810 23:31:21.031098 18574 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.125:35865: Network error: Client connection negotiation failed: client connection to 127.14.175.125:35865: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.125.keytab.
Entry for principal kudu/127.14.175.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.125.keytab.
I20260810 23:31:21.068753 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.125:35865
--webserver_interface=127.14.175.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.125.keytab
--principal=kudu/127.14.175.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:37981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:37997,127.14.175.125:35865,127.14.175.124:42283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:31:21.080540 18574 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.125" port: 35865 } attempt: 1
I20260810 23:31:21.081874 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.081358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:35865 (local address 127.0.0.1:40102)
0810 23:31:21.081578 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:21.081701 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.125:35865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 23:31:21.083101 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.082644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:35865 (local address 127.0.0.1:40112)
0810 23:31:21.082822 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:21.082921 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.125:35865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 23:31:21.083345 18574 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.125:35865: Network error: Client connection negotiation failed: client connection to 127.14.175.125:35865: connect: Connection refused (error 111)
I20260810 23:31:21.151648 18574 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.125" port: 35865 } attempt: 2
I20260810 23:31:21.153033 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.152420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:35865 (local address 127.0.0.1:40114)
0810 23:31:21.152712 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:21.152849 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.125:35865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260810 23:31:21.154258 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.153803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:35865 (local address 127.0.0.1:40130)
0810 23:31:21.153953 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:21.154070 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.125:35865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 23:31:21.154536 18574 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.125:35865: Network error: Client connection negotiation failed: client connection to 127.14.175.125:35865: connect: Connection refused (error 111)
I20260810 23:31:21.245858 18574 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.125" port: 35865 } attempt: 3
I20260810 23:31:21.247164 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.246629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:35865 (local address 127.0.0.1:40134)
0810 23:31:21.246860 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:21.246982 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.125:35865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260810 23:31:21.248495 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.247972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:35865 (local address 127.0.0.1:40140)
0810 23:31:21.248164 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:21.248273 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.125:35865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260810 23:31:21.248786 18574 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.125:35865: Network error: Client connection negotiation failed: client connection to 127.14.175.125:35865: connect: Connection refused (error 111)
I20260810 23:31:21.392094 18574 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.125" port: 35865 } attempt: 4
I20260810 23:31:21.393600 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.392884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:35865 (local address 127.0.0.1:40142)
0810 23:31:21.393276 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:21.393394 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.125:35865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260810 23:31:21.394784 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.394352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:35865 (local address 127.0.0.1:40158)
0810 23:31:21.394525 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:21.394618 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.125:35865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 23:31:21.394980 18574 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.125:35865: Network error: Client connection negotiation failed: client connection to 127.14.175.125:35865: connect: Connection refused (error 111)
W20260810 23:31:21.437040 18584 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:31:21.437465 18584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:31:21.437561 18584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:31:21.437647 18584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:31:21.445264 18584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:31:21.445364 18584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:31:21.445421 18584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:31:21.445472 18584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:31:21.445536 18584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:31:21.455895 18584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:37981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:37997,127.14.175.125:35865,127.14.175.124:42283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.125.keytab
--principal=kudu/127.14.175.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.125:35865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.175.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:31:21.457302 18584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:31:21.458663 18584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:31:21.470780 18594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:21.470765 18595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:21.470958 18599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:31:21.471693 18584 server_base.cc:1034] running on GCE node
I20260810 23:31:21.472556 18584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:31:21.474066 18584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:31:21.475286 18584 hybrid_clock.cc:648] HybridClock initialized: now 1786404681475207 us; error 58 us; skew 500 ppm
I20260810 23:31:21.475615 18584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:31:21 dist-test-slave-rmhg krb5kdc[18437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404681, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:31:21.478830 18584 init.cc:377] Logged in from keytab as kudu/127.14.175.125@KRBTEST.COM (short username kudu)
I20260810 23:31:21.480309 18584 webserver.cc:460] Webserver started at http://127.14.175.125:42077/ using document root <none> and password file <none>
I20260810 23:31:21.480998 18584 fs_manager.cc:359] Metadata directory not provided
I20260810 23:31:21.481087 18584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:31:21.481416 18584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:31:21.484424 18584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-1/data/instance:
uuid: "c2ebaaa659cc464282dcd46444f1024e"
format_stamp: "Formatted at 2026-08-10 23:31:21 on dist-test-slave-rmhg"
server_key: "fb07a8bda3f36216395de4bef5be1068"
server_key_iv: "f4459148e3842bb35f64f63a114a606b"
server_key_version: "encryptionkey@0"
I20260810 23:31:21.485262 18584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-1/wal/instance:
uuid: "c2ebaaa659cc464282dcd46444f1024e"
format_stamp: "Formatted at 2026-08-10 23:31:21 on dist-test-slave-rmhg"
server_key: "fb07a8bda3f36216395de4bef5be1068"
server_key_iv: "f4459148e3842bb35f64f63a114a606b"
server_key_version: "encryptionkey@0"
I20260810 23:31:21.490734 18584 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 23:31:21.494444 18609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:21.495882 18584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:31:21.496037 18584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-1/wal
uuid: "c2ebaaa659cc464282dcd46444f1024e"
format_stamp: "Formatted at 2026-08-10 23:31:21 on dist-test-slave-rmhg"
server_key: "fb07a8bda3f36216395de4bef5be1068"
server_key_iv: "f4459148e3842bb35f64f63a114a606b"
server_key_version: "encryptionkey@0"
I20260810 23:31:21.496249 18584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:31:21.512534 18584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:31:21.516700 18584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:31:21.516943 18584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:31:21.535436 18584 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.125:35865
I20260810 23:31:21.535456 18694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.125:35865 every 8 connection(s)
I20260810 23:31:21.536921 18584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-1/data/info.pb
I20260810 23:31:21.541033 18695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:21.544057 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 18584
I20260810 23:31:21.544265 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-1/wal/instance
I20260810 23:31:21.544708 15037 external_mini_cluster.cc:1428] Setting key d12d829789d9483c1377ce94df943a42
I20260810 23:31:21.548096 18695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37997 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:31:21 dist-test-slave-rmhg krb5kdc[18437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404681, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.125@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:31:21.569136 18698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.550964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55098 (local address 127.14.175.126:37997)
0810 23:31:21.551443 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:21.551448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:21.552703 (+ 1255us) server_negotiation.cc:407] Connection header received
0810 23:31:21.554759 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:21.554765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:21.554850 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:21.554950 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:21.556655 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:21.557443 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:21.559039 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:21.559469 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:21.559586 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:21.563179 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:21.563184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:21.563188 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:21.563235 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:21.565906 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:21.566699 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:21.566704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:21.566706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:21.566841 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:21.567280 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:21.567286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:21.567288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:21.567593 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:21.567737 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:21.568504 (+ 767us) server_negotiation.cc:299] Negotiation successful
0810 23:31:21.568752 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":257,"threads_started":1}
I20260810 23:31:21.570227 18700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.551693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:37997 (local address 127.0.0.1:55098)
0810 23:31:21.552512 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:21.552552 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:21.554436 (+ 1884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:21.555227 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:21.555237 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:21.555815 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:21.556423 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:21.556442 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:21.557624 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:21.557628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:21.558872 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:21.558891 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:21.559719 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:21.559727 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:21.559970 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:21.560637 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:21.560662 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:21.562931 (+ 2269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:21.566062 (+ 3131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:21.566073 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:21.566094 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:21.566562 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:21.566956 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:21.566962 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:21.566966 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:21.567165 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:21.567731 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:21.567740 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:21.568119 (+ 379us) client_negotiation.cc:769] Sending connection context
0810 23:31:21.568383 (+ 264us) client_negotiation.cc:240] Negotiation successful
0810 23:31:21.568958 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":335,"threads_started":1}
I20260810 23:31:21.572001 18695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.125" port: 35865 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.175.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.124@KRBTEST.COM" created.
Aug 10 23:31:21 dist-test-slave-rmhg krb5kdc[18437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404681, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.125@KRBTEST.COM for kudu/127.14.175.125@KRBTEST.COM
I20260810 23:31:21.588479 18700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.572668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:35865 (local address 127.0.0.1:40172)
0810 23:31:21.572928 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:21.572944 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:21.573032 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:21.574222 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:21.574226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:21.574682 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:21.574828 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:21.574837 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:21.576414 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:21.576419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:21.577520 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:21.577535 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:21.578592 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:21.578598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:21.578792 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:21.579419 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:21.579442 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:21.581923 (+ 2481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:21.585813 (+ 3890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:21.585818 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:21.585820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:21.586249 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:21.586709 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:21.586713 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:21.586716 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:21.586876 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:21.587598 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:21.587602 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:21.587742 (+ 140us) client_negotiation.cc:769] Sending connection context
0810 23:31:21.587859 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 23:31:21.588133 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260810 23:31:21.588728 18703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.573002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40172 (local address 127.14.175.125:35865)
0810 23:31:21.573791 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:21.573797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:21.573824 (+ 27us) server_negotiation.cc:407] Connection header received
0810 23:31:21.573870 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:21.573877 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:21.574069 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:21.574217 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:21.575064 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:21.576246 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:21.577676 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:21.578413 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:21.578519 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:21.582181 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:21.582201 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:21.582223 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:21.582261 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:21.585638 (+ 3377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:21.586399 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:21.586409 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:21.586412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:21.586567 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:21.586978 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:21.586982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:21.586985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:21.587475 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:21.587703 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:21.588029 (+ 326us) server_negotiation.cc:299] Negotiation successful
0810 23:31:21.588304 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":479,"threads_started":1}
I20260810 23:31:21.591585 18695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42283 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:31:21.592849 18700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.592377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:42283 (local address 127.0.0.1:52708)
0810 23:31:21.592545 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:21.592673 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:42283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260810 23:31:21.593209 18614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.175.124:42283: connect: Connection refused (error 111)
I20260810 23:31:21.594816 18700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.594459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:42283 (local address 127.0.0.1:52712)
0810 23:31:21.594618 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:21.594705 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:42283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260810 23:31:21.595046 18695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.124:42283: Network error: Client connection negotiation failed: client connection to 127.14.175.124:42283: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.124.keytab.
Entry for principal kudu/127.14.175.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.124.keytab.
I20260810 23:31:21.618003 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.124:42283
--webserver_interface=127.14.175.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.124.keytab
--principal=kudu/127.14.175.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:37981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:37997,127.14.175.125:35865,127.14.175.124:42283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:31:21.644380 18695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42283 } attempt: 1
I20260810 23:31:21.645747 18700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.645252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:42283 (local address 127.0.0.1:52714)
0810 23:31:21.645448 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:21.645559 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:42283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 23:31:21.646811 18700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.646451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:42283 (local address 127.0.0.1:52722)
0810 23:31:21.646622 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:21.646700 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:42283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 23:31:21.647022 18695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.124:42283: Network error: Client connection negotiation failed: client connection to 127.14.175.124:42283: connect: Connection refused (error 111)
I20260810 23:31:21.694262 18574 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.125" port: 35865 } attempt: 5
Aug 10 23:31:21 dist-test-slave-rmhg krb5kdc[18437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404680, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for kudu/127.14.175.125@KRBTEST.COM
I20260810 23:31:21.708618 18703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.695144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40182 (local address 127.14.175.125:35865)
0810 23:31:21.695347 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:21.695353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:21.695453 (+ 100us) server_negotiation.cc:407] Connection header received
0810 23:31:21.695660 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:21.695665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:21.695753 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:21.695843 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:21.696693 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:21.697467 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:21.698716 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:21.699190 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:21.699335 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:21.703061 (+ 3726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:21.703067 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:21.703070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:21.703109 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:21.705848 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:21.706583 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:21.706588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:21.706590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:21.706732 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:21.707127 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:21.707132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:21.707134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:21.707475 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:21.707629 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:21.708027 (+ 398us) server_negotiation.cc:299] Negotiation successful
0810 23:31:21.708256 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 23:31:21.708606 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.695081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:35865 (local address 127.0.0.1:40182)
0810 23:31:21.695347 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:21.695368 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:21.695518 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:21.695909 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:21.695916 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:21.696346 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:21.696516 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:21.696526 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:21.697605 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:21.697610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:21.698551 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:21.698565 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:21.699347 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:21.699352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:21.699559 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:21.700316 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:21.700343 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:21.702840 (+ 2497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:21.706016 (+ 3176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:21.706023 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:21.706028 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:21.706454 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:21.706854 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:21.706859 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:21.706863 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:21.707013 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:21.707610 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:21.707615 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:21.707774 (+ 159us) client_negotiation.cc:769] Sending connection context
0810 23:31:21.707902 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 23:31:21.708270 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118}
I20260810 23:31:21.710047 18574 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42283 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:31:21.711263 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.710855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:42283 (local address 127.0.0.1:52734)
0810 23:31:21.711025 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:21.711125 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:42283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 23:31:21.712205 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.711855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:42283 (local address 127.0.0.1:52744)
0810 23:31:21.711996 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:21.712084 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:42283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 23:31:21.712396 18574 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.124:42283: Network error: Client connection negotiation failed: client connection to 127.14.175.124:42283: connect: Connection refused (error 111)
I20260810 23:31:21.715220 18695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42283 } attempt: 2
I20260810 23:31:21.716248 18700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.715844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:42283 (local address 127.0.0.1:52748)
0810 23:31:21.716009 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:21.716094 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:42283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 23:31:21.717304 18700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.716922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:42283 (local address 127.0.0.1:52760)
0810 23:31:21.717052 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:21.717143 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:42283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 23:31:21.717505 18695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.124:42283: Network error: Client connection negotiation failed: client connection to 127.14.175.124:42283: connect: Connection refused (error 111)
I20260810 23:31:21.763652 18574 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42283 } attempt: 1
I20260810 23:31:21.764825 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.764327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:42283 (local address 127.0.0.1:52770)
0810 23:31:21.764545 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:21.764659 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:42283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260810 23:31:21.765974 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.765647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:42283 (local address 127.0.0.1:52774)
0810 23:31:21.765777 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:21.765854 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:42283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 23:31:21.766206 18574 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.124:42283: Network error: Client connection negotiation failed: client connection to 127.14.175.124:42283: connect: Connection refused (error 111)
I20260810 23:31:21.808761 18695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42283 } attempt: 3
I20260810 23:31:21.810082 18700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.809541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:42283 (local address 127.0.0.1:52784)
0810 23:31:21.809802 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:21.809910 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:42283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260810 23:31:21.811430 18700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.810910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:42283 (local address 127.0.0.1:52800)
0810 23:31:21.811191 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:21.811283 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:42283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
W20260810 23:31:21.811779 18695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.124:42283: Network error: Client connection negotiation failed: client connection to 127.14.175.124:42283: connect: Connection refused (error 111)
I20260810 23:31:21.834481 18574 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42283 } attempt: 2
I20260810 23:31:21.835913 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.835306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:42283 (local address 127.0.0.1:52810)
0810 23:31:21.835607 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:21.835723 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:42283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260810 23:31:21.837033 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.836680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:42283 (local address 127.0.0.1:52822)
0810 23:31:21.836830 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:21.836910 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:42283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 23:31:21.837316 18574 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.124:42283: Network error: Client connection negotiation failed: client connection to 127.14.175.124:42283: connect: Connection refused (error 111)
I20260810 23:31:21.943534 18574 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42283 } attempt: 3
I20260810 23:31:21.944746 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.944253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:42283 (local address 127.0.0.1:52830)
0810 23:31:21.944441 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:21.944544 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:42283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 23:31:21.945822 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.945445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:42283 (local address 127.0.0.1:52832)
0810 23:31:21.945644 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:21.945715 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:42283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260810 23:31:21.946050 18574 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.124:42283: Network error: Client connection negotiation failed: client connection to 127.14.175.124:42283: connect: Connection refused (error 111)
I20260810 23:31:21.954979 18695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42283 } attempt: 4
I20260810 23:31:21.956131 18700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.955667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:42283 (local address 127.0.0.1:52844)
0810 23:31:21.955882 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:21.955973 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:42283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 23:31:21.957175 18700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:21.956826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:42283 (local address 127.0.0.1:52850)
0810 23:31:21.956961 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:21.957033 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:42283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 23:31:21.957393 18695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.124:42283: Network error: Client connection negotiation failed: client connection to 127.14.175.124:42283: connect: Connection refused (error 111)
W20260810 23:31:21.985505 18707 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:31:21.985903 18707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:31:21.986009 18707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:31:21.986123 18707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:31:21.993774 18707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:31:21.993877 18707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:31:21.993968 18707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:31:21.994028 18707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:31:21.994095 18707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:31:22.005283 18707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:37981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:37997,127.14.175.125:35865,127.14.175.124:42283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.124.keytab
--principal=kudu/127.14.175.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.124:42283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.175.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:31:22.006763 18707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:31:22.008229 18707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:31:22.020418 18721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:22.020370 18719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:22.020447 18715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:31:22.021441 18707 server_base.cc:1034] running on GCE node
I20260810 23:31:22.022317 18707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:31:22.023775 18707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:31:22.025008 18707 hybrid_clock.cc:648] HybridClock initialized: now 1786404682024938 us; error 45 us; skew 500 ppm
I20260810 23:31:22.025398 18707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:31:22 dist-test-slave-rmhg krb5kdc[18437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404682, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:31:22.028604 18707 init.cc:377] Logged in from keytab as kudu/127.14.175.124@KRBTEST.COM (short username kudu)
I20260810 23:31:22.030198 18707 webserver.cc:460] Webserver started at http://127.14.175.124:46261/ using document root <none> and password file <none>
I20260810 23:31:22.030913 18707 fs_manager.cc:359] Metadata directory not provided
I20260810 23:31:22.031015 18707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:31:22.031345 18707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:31:22.034477 18707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-2/data/instance:
uuid: "571a64b9bc554f4892b21227290e43b5"
format_stamp: "Formatted at 2026-08-10 23:31:22 on dist-test-slave-rmhg"
server_key: "f0d86094a17fbb5113f53a5ff65c8c61"
server_key_iv: "218874103a60b2306eec6eccae9c1612"
server_key_version: "encryptionkey@0"
I20260810 23:31:22.035274 18707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-2/wal/instance:
uuid: "571a64b9bc554f4892b21227290e43b5"
format_stamp: "Formatted at 2026-08-10 23:31:22 on dist-test-slave-rmhg"
server_key: "f0d86094a17fbb5113f53a5ff65c8c61"
server_key_iv: "218874103a60b2306eec6eccae9c1612"
server_key_version: "encryptionkey@0"
I20260810 23:31:22.040545 18707 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:31:22.044232 18733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:22.045677 18707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:31:22.045835 18707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-2/wal
uuid: "571a64b9bc554f4892b21227290e43b5"
format_stamp: "Formatted at 2026-08-10 23:31:22 on dist-test-slave-rmhg"
server_key: "f0d86094a17fbb5113f53a5ff65c8c61"
server_key_iv: "218874103a60b2306eec6eccae9c1612"
server_key_version: "encryptionkey@0"
I20260810 23:31:22.046049 18707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:31:22.060556 18707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:31:22.065253 18707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:31:22.065529 18707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:31:22.083796 18707 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.124:42283
I20260810 23:31:22.083813 18821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.124:42283 every 8 connection(s)
I20260810 23:31:22.085364 18707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-2/data/info.pb
I20260810 23:31:22.089193 18822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:22.093613 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 18707
I20260810 23:31:22.093822 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404299514076-15037-0/minicluster-data/master-2/wal/instance
I20260810 23:31:22.094381 15037 external_mini_cluster.cc:1428] Setting key daf24abe8b55917b39df1075dc76a64b
I20260810 23:31:22.096786 18822 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37997 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:31:22 dist-test-slave-rmhg krb5kdc[18437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404682, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.124@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
Aug 10 23:31:22 dist-test-slave-rmhg krb5kdc[18437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.124@KRBTEST.COM
I20260810 23:31:22.115828 18831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:22.099618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55106 (local address 127.14.175.126:37997)
0810 23:31:22.100992 (+ 1374us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:22.100998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:22.101018 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:31:22.101116 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:22.101121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:22.101237 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:22.101357 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:22.102685 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:22.103424 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:22.105130 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:22.105599 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:22.105709 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:22.109629 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:22.109637 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:22.109643 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:22.109690 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:22.112464 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:22.113336 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:22.113343 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:22.113345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:22.113486 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:22.113935 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:22.113939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:22.113941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:22.114234 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:22.114385 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:22.115171 (+ 786us) server_negotiation.cc:299] Negotiation successful
0810 23:31:22.115418 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1252,"thread_start_us":1114,"threads_started":1}
I20260810 23:31:22.116941 18829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:22.096462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52854 (local address 127.14.175.124:42283)
0810 23:31:22.097354 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:22.097370 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:22.097417 (+ 47us) server_negotiation.cc:407] Connection header received
0810 23:31:22.099175 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:22.099200 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:22.099681 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:22.100182 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:22.102517 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:22.103851 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:22.105228 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:22.105867 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:22.106136 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:22.110176 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:22.110187 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:22.110206 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:22.110245 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:22.113088 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:22.113830 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:22.113837 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:22.113839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:22.113973 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:22.114357 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:22.114360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:22.114362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:22.114787 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:22.115108 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:22.115447 (+ 339us) server_negotiation.cc:299] Negotiation successful
0810 23:31:22.115703 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":627,"thread_start_us":395,"threads_started":1}
I20260810 23:31:22.116982 18830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:22.099504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:37997 (local address 127.0.0.1:55106)
0810 23:31:22.100246 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:22.100374 (+ 128us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:22.100593 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:22.101461 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:22.101466 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:22.102062 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:22.102467 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:22.102494 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:22.103585 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:22.103591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:22.104945 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:22.104959 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:22.105839 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:22.105847 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:22.106096 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:22.106939 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:22.106962 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:22.109387 (+ 2425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:22.112622 (+ 3235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:22.112636 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:22.112651 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:22.113215 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:22.113609 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:22.113614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:22.113618 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:22.113819 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:22.114363 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:22.114372 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:22.114783 (+ 411us) client_negotiation.cc:769] Sending connection context
0810 23:31:22.115047 (+ 264us) client_negotiation.cc:240] Negotiation successful
0810 23:31:22.115581 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":70,"spinlock_wait_cycles":18176,"thread_start_us":339,"threads_started":1}
I20260810 23:31:22.119771 18822 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.125" port: 35865 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:31:22.123266 18574 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42283 } attempt: 4
Aug 10 23:31:22 dist-test-slave-rmhg krb5kdc[18437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404682, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.124@KRBTEST.COM for kudu/127.14.175.125@KRBTEST.COM
Aug 10 23:31:22 dist-test-slave-rmhg krb5kdc[18437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
Aug 10 23:31:22 dist-test-slave-rmhg krb5kdc[18437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404680, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for kudu/127.14.175.124@KRBTEST.COM
I20260810 23:31:22.135115 18830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:22.120386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:35865 (local address 127.0.0.1:40184)
0810 23:31:22.120714 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:22.120732 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:22.120842 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:22.121141 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:22.121145 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:22.121559 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:22.121724 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:22.121734 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:22.122713 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:22.122717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:22.123721 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:22.123735 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:22.124809 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:22.124815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:22.124997 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:22.126486 (+ 1489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:22.126503 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:22.128741 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:22.132275 (+ 3534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:22.132281 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:22.132286 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:22.132823 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:22.133362 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:22.133367 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:22.133371 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:22.133526 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:22.134064 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:22.134068 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:22.134230 (+ 162us) client_negotiation.cc:769] Sending connection context
0810 23:31:22.134389 (+ 159us) client_negotiation.cc:240] Negotiation successful
0810 23:31:22.134761 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216}
I20260810 23:31:22.135140 18703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:22.120682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40184 (local address 127.14.175.125:35865)
0810 23:31:22.120860 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:22.120865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:22.120883 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:31:22.120933 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:22.120937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:22.120995 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:22.121084 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:22.121878 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:22.122566 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:22.123918 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:22.124582 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:22.124700 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:22.128953 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:22.128959 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:22.128962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:22.128995 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:22.132137 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:22.132963 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:22.132969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:22.132973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:22.133214 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:22.133631 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:22.133637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:22.133640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:22.133930 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:22.134078 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:22.134498 (+ 420us) server_negotiation.cc:299] Negotiation successful
0810 23:31:22.134731 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 23:31:22.136479 18822 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42283 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:31:22.137347 18831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:22.120386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55110 (local address 127.14.175.126:37997)
0810 23:31:22.121024 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:22.121028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:22.121041 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:31:22.121082 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:22.121085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:22.121178 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:22.121272 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:22.122570 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:22.123350 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:22.124911 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:22.125552 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:22.125678 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:22.130682 (+ 5004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:22.130689 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:22.130694 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:22.130735 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:22.133986 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:22.134949 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:22.134954 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:22.134958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:22.135118 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:22.135615 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:22.135620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:22.135624 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:22.136011 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:22.136295 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:22.136705 (+ 410us) server_negotiation.cc:299] Negotiation successful
0810 23:31:22.136947 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527}
I20260810 23:31:22.139374 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:22.123873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:42283 (local address 127.0.0.1:52864)
0810 23:31:22.124073 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:22.124094 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:22.124206 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:22.124673 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:22.124679 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:22.125141 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:22.125399 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:22.125411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:22.126429 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:22.126433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:22.127837 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:22.127855 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:22.128838 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:22.128845 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:22.129059 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:22.129641 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:22.129665 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:22.132548 (+ 2883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:22.136333 (+ 3785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:22.136339 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:22.136343 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:22.136838 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:22.137366 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:22.137371 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:22.137375 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:22.137591 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:22.138288 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:22.138294 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:22.138500 (+ 206us) client_negotiation.cc:769] Sending connection context
0810 23:31:22.138644 (+ 144us) client_negotiation.cc:240] Negotiation successful
0810 23:31:22.138964 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260810 23:31:22.140591 18829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:22.124209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52864 (local address 127.14.175.124:42283)
0810 23:31:22.124370 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:22.124376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:22.124393 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:31:22.124437 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:22.124441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:22.124527 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:22.124639 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:22.125549 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:22.126300 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:22.127998 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:22.128660 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:22.128784 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:22.132775 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:22.132781 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:22.132785 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:22.132820 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:22.136191 (+ 3371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:22.137001 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:22.137006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:22.137009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:22.137231 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:22.137743 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:22.137748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:22.137751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:22.138130 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:22.139811 (+ 1681us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:22.139958 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 23:31:22.140237 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 23:31:22.147511 18574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31f9bdd654f047bb8c98a5e05e8bff7c: Bootstrap starting.
Aug 10 23:31:22 dist-test-slave-rmhg krb5kdc[18437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404682, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.124@KRBTEST.COM for kudu/127.14.175.124@KRBTEST.COM
Aug 10 23:31:22 dist-test-slave-rmhg krb5kdc[18437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.125@KRBTEST.COM
I20260810 23:31:22.152292 18574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31f9bdd654f047bb8c98a5e05e8bff7c: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:22.153919 18574 log.cc:824] T 00000000000000000000000000000000 P 31f9bdd654f047bb8c98a5e05e8bff7c: Log is configured to *not* fsync() on all Append() calls
I20260810 23:31:22.157503 18574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31f9bdd654f047bb8c98a5e05e8bff7c: No bootstrap required, opened a new log
I20260810 23:31:22.158578 18830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:22.137385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:42283 (local address 127.0.0.1:52874)
0810 23:31:22.138288 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:22.138307 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:22.138400 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:22.141690 (+ 3290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:22.141695 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:22.142408 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:22.142633 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:22.142646 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:22.144094 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:22.144100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:22.145645 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:22.145662 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:22.146903 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:22.146909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:22.147138 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:22.147828 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:22.147854 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:22.151797 (+ 3943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:22.155571 (+ 3774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:22.155577 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:22.155580 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:22.156059 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:22.156516 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:22.156521 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:22.156525 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:22.156686 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:22.157507 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:22.157511 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:22.157665 (+ 154us) client_negotiation.cc:769] Sending connection context
0810 23:31:22.157821 (+ 156us) client_negotiation.cc:240] Negotiation successful
0810 23:31:22.158179 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779}
I20260810 23:31:22.158624 18703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:22.139009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40192 (local address 127.14.175.125:35865)
0810 23:31:22.139173 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:22.139178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:22.139239 (+ 61us) server_negotiation.cc:407] Connection header received
0810 23:31:22.139444 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:22.139450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:22.139526 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:22.139591 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:22.141003 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:22.142076 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:22.143838 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:22.145519 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:22.145630 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:22.152238 (+ 6608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:22.152246 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:22.152250 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:22.152296 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:22.155735 (+ 3439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:22.156565 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:22.156571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:22.156575 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:22.156732 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:22.157112 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:22.157117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:22.157121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:22.157441 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:22.157715 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:22.158028 (+ 313us) server_negotiation.cc:299] Negotiation successful
0810 23:31:22.158265 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:31:22.158761 18829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:22.137901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52874 (local address 127.14.175.124:42283)
0810 23:31:22.141284 (+ 3383us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:22.141290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:22.141309 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:31:22.141365 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:22.141371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:22.141442 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:22.141538 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:22.142791 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:22.143933 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:22.145815 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:22.146548 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:22.146670 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:22.152066 (+ 5396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:22.152074 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:22.152079 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:22.152119 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:22.155395 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:22.156222 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:22.156228 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:22.156231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:22.156382 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:22.156808 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:22.156813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:22.156816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:22.157292 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:22.157444 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:22.157976 (+ 532us) server_negotiation.cc:299] Negotiation successful
0810 23:31:22.158344 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":125,"server-negotiator.queue_time_us":3224,"thread_start_us":15074,"threads_started":1}
I20260810 23:31:22.164065 18822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 571a64b9bc554f4892b21227290e43b5: Bootstrap starting.
I20260810 23:31:22.166265 15037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:31:22.167753 18822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 571a64b9bc554f4892b21227290e43b5: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:22.167555 18574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31f9bdd654f047bb8c98a5e05e8bff7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f9bdd654f047bb8c98a5e05e8bff7c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37997 } } peers { permanent_uuid: "c2ebaaa659cc464282dcd46444f1024e" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 35865 } } peers { permanent_uuid: "571a64b9bc554f4892b21227290e43b5" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42283 } }
I20260810 23:31:22.167951 18574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31f9bdd654f047bb8c98a5e05e8bff7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f9bdd654f047bb8c98a5e05e8bff7c, State: Initialized, Role: FOLLOWER
I20260810 23:31:22.168442 18574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31f9bdd654f047bb8c98a5e05e8bff7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f9bdd654f047bb8c98a5e05e8bff7c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37997 } } peers { permanent_uuid: "c2ebaaa659cc464282dcd46444f1024e" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 35865 } } peers { permanent_uuid: "571a64b9bc554f4892b21227290e43b5" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42283 } }
I20260810 23:31:22.168762 18822 log.cc:824] T 00000000000000000000000000000000 P 571a64b9bc554f4892b21227290e43b5: Log is configured to *not* fsync() on all Append() calls
I20260810 23:31:22.169751 18533 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "31f9bdd654f047bb8c98a5e05e8bff7c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55110
I20260810 23:31:22.170698 18835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31f9bdd654f047bb8c98a5e05e8bff7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f9bdd654f047bb8c98a5e05e8bff7c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37997 } } peers { permanent_uuid: "c2ebaaa659cc464282dcd46444f1024e" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 35865 } } peers { permanent_uuid: "571a64b9bc554f4892b21227290e43b5" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42283 } } }
I20260810 23:31:22.171167 18835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31f9bdd654f047bb8c98a5e05e8bff7c [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:31:22.171794 18574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31f9bdd654f047bb8c98a5e05e8bff7c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:31:22.172019 18533 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31f9bdd654f047bb8c98a5e05e8bff7c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 23:31:22.172245 18533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31f9bdd654f047bb8c98a5e05e8bff7c [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:22.173111 18822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 571a64b9bc554f4892b21227290e43b5: No bootstrap required, opened a new log
I20260810 23:31:22.175839 18533 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31f9bdd654f047bb8c98a5e05e8bff7c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f9bdd654f047bb8c98a5e05e8bff7c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37997 } } peers { permanent_uuid: "c2ebaaa659cc464282dcd46444f1024e" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 35865 } } peers { permanent_uuid: "571a64b9bc554f4892b21227290e43b5" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42283 } }
I20260810 23:31:22.177851 18533 leader_election.cc:288] T 00000000000000000000000000000000 P 31f9bdd654f047bb8c98a5e05e8bff7c [CANDIDATE]: Term 1 election: Requested vote from peers c2ebaaa659cc464282dcd46444f1024e (127.14.175.125:35865), 571a64b9bc554f4892b21227290e43b5 (127.14.175.124:42283)
I20260810 23:31:22.185539 18822 raft_consensus.cc:348] T 00000000000000000000000000000000 P 571a64b9bc554f4892b21227290e43b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f9bdd654f047bb8c98a5e05e8bff7c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37997 } } peers { permanent_uuid: "c2ebaaa659cc464282dcd46444f1024e" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 35865 } } peers { permanent_uuid: "571a64b9bc554f4892b21227290e43b5" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42283 } }
I20260810 23:31:22.186033 18822 raft_consensus.cc:729] T 00000000000000000000000000000000 P 571a64b9bc554f4892b21227290e43b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 571a64b9bc554f4892b21227290e43b5, State: Initialized, Role: FOLLOWER
I20260810 23:31:22.186645 18822 consensus_queue.cc:260] T 00000000000000000000000000000000 P 571a64b9bc554f4892b21227290e43b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f9bdd654f047bb8c98a5e05e8bff7c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37997 } } peers { permanent_uuid: "c2ebaaa659cc464282dcd46444f1024e" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 35865 } } peers { permanent_uuid: "571a64b9bc554f4892b21227290e43b5" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42283 } }
I20260810 23:31:22.188040 18851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 571a64b9bc554f4892b21227290e43b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f9bdd654f047bb8c98a5e05e8bff7c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37997 } } peers { permanent_uuid: "c2ebaaa659cc464282dcd46444f1024e" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 35865 } } peers { permanent_uuid: "571a64b9bc554f4892b21227290e43b5" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42283 } } }
I20260810 23:31:22.189083 18851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 571a64b9bc554f4892b21227290e43b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:31:22.189529 18822 sys_catalog.cc:564] T 00000000000000000000000000000000 P 571a64b9bc554f4892b21227290e43b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:31:22.189628 18781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31f9bdd654f047bb8c98a5e05e8bff7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "571a64b9bc554f4892b21227290e43b5"
I20260810 23:31:22.190475 18781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 571a64b9bc554f4892b21227290e43b5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:22.193960 18781 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 571a64b9bc554f4892b21227290e43b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f9bdd654f047bb8c98a5e05e8bff7c in term 1.
I20260810 23:31:22.194695 18494 leader_election.cc:302] T 00000000000000000000000000000000 P 31f9bdd654f047bb8c98a5e05e8bff7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f9bdd654f047bb8c98a5e05e8bff7c, 571a64b9bc554f4892b21227290e43b5; no voters:
I20260810 23:31:22.195322 18835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31f9bdd654f047bb8c98a5e05e8bff7c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:31:22.207298 18835 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31f9bdd654f047bb8c98a5e05e8bff7c [term 1 LEADER]: Becoming Leader. State: Replica: 31f9bdd654f047bb8c98a5e05e8bff7c, State: Running, Role: LEADER
I20260810 23:31:22.208164 18835 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31f9bdd654f047bb8c98a5e05e8bff7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f9bdd654f047bb8c98a5e05e8bff7c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37997 } } peers { permanent_uuid: "c2ebaaa659cc464282dcd46444f1024e" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 35865 } } peers { permanent_uuid: "571a64b9bc554f4892b21227290e43b5" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42283 } }
I20260810 23:31:22.219300 18831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:22.187669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55118 (local address 127.14.175.126:37997)
0810 23:31:22.187865 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:22.187872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:22.196561 (+ 8689us) server_negotiation.cc:407] Connection header received
0810 23:31:22.197270 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:22.197275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:22.197350 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:22.197413 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:22.201798 (+ 4385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:22.202956 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:22.204790 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:22.205516 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:22.205643 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:22.211955 (+ 6312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:22.211963 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:22.211968 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:22.212020 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:22.215701 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:22.216596 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:22.216602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:22.216606 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:22.216780 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:22.217368 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:22.217374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:22.217377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:22.217803 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:22.218078 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:22.218656 (+ 578us) server_negotiation.cc:299] Negotiation successful
0810 23:31:22.218901 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260810 23:31:22.219738 18502 tablet.cc:2369] T 00000000000000000000000000000000 P 31f9bdd654f047bb8c98a5e05e8bff7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:31:22.220664 18703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:22.191333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40196 (local address 127.14.175.125:35865)
0810 23:31:22.191511 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:22.191516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:22.196888 (+ 5372us) server_negotiation.cc:407] Connection header received
0810 23:31:22.197204 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:22.197210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:22.197297 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:22.197366 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:22.199687 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:22.200777 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:22.207272 (+ 6495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:22.207915 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:22.208054 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:22.212382 (+ 4328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:22.212389 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:22.212393 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:22.212438 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:22.216521 (+ 4083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:22.217526 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:22.217532 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:22.217536 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:22.217717 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:22.218209 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:22.218215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:22.218219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:22.218625 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:22.218871 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:22.219981 (+ 1110us) server_negotiation.cc:299] Negotiation successful
0810 23:31:22.220280 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 23:31:22.225548 18862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31f9bdd654f047bb8c98a5e05e8bff7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31f9bdd654f047bb8c98a5e05e8bff7c. Latest consensus state: current_term: 1 leader_uuid: "31f9bdd654f047bb8c98a5e05e8bff7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f9bdd654f047bb8c98a5e05e8bff7c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37997 } } peers { permanent_uuid: "c2ebaaa659cc464282dcd46444f1024e" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 35865 } } peers { permanent_uuid: "571a64b9bc554f4892b21227290e43b5" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42283 } } }
I20260810 23:31:22.226713 18862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31f9bdd654f047bb8c98a5e05e8bff7c [sys.catalog]: This master's current role is: LEADER
I20260810 23:31:22.227339 18829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:22.188117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52882 (local address 127.14.175.124:42283)
0810 23:31:22.188294 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:22.188300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:22.190487 (+ 2187us) server_negotiation.cc:407] Connection header received
0810 23:31:22.192492 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:22.192499 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:22.192664 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:22.192752 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:22.199628 (+ 6876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:22.200726 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:22.206170 (+ 5444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:22.206913 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:22.207014 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:22.214060 (+ 7046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:22.214068 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:22.214073 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:22.214120 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:22.218833 (+ 4713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:22.220204 (+ 1371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:22.220211 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:22.220215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:22.220480 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:22.221506 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:22.221539 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:22.221543 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:22.223475 (+ 1932us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:22.223752 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:22.226697 (+ 2945us) server_negotiation.cc:299] Negotiation successful
0810 23:31:22.226989 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 23:31:22.226028 18871 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 23:31:22.238471 18873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:31:22.230788 18459 master_runner.cc:186] Error getting master registration for 127.14.175.125:35865: OK, instance_id { permanent_uuid: "c2ebaaa659cc464282dcd46444f1024e" instance_seqno: 1786404681530222 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:31:22.245524 18459 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:31:22.246537 18873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:31:22.256599 18695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42283 } attempt: 5
I20260810 23:31:22.261376 18781 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 571a64b9bc554f4892b21227290e43b5 [term 1 FOLLOWER]: Refusing update from remote peer 31f9bdd654f047bb8c98a5e05e8bff7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:31:22.262533 18862 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 31f9bdd654f047bb8c98a5e05e8bff7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "571a64b9bc554f4892b21227290e43b5" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Aug 10 23:31:22 dist-test-slave-rmhg krb5kdc[18437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404681, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.125@KRBTEST.COM for kudu/127.14.175.124@KRBTEST.COM
I20260810 23:31:22.274060 18700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:22.257358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:42283 (local address 127.0.0.1:52886)
0810 23:31:22.257554 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:22.257575 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:22.257722 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:22.258101 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:22.258107 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:22.258652 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:22.258886 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:22.258899 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:22.260559 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:22.260569 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:22.261785 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:22.261802 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:22.262781 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:22.262787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:22.263008 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:22.263698 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:22.263721 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:22.266573 (+ 2852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:22.270905 (+ 4332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:22.270910 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:22.270913 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:22.271426 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:22.271906 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:22.271911 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:22.271914 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:22.272177 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:22.272957 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:22.272963 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:22.273138 (+ 175us) client_negotiation.cc:769] Sending connection context
0810 23:31:22.273310 (+ 172us) client_negotiation.cc:240] Negotiation successful
0810 23:31:22.273654 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 23:31:22.275215 18829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:22.257441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52886 (local address 127.14.175.124:42283)
0810 23:31:22.257598 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:22.257603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:22.257681 (+ 78us) server_negotiation.cc:407] Connection header received
0810 23:31:22.257870 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:22.257876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:22.257966 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:22.258065 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:22.259044 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:22.260255 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:22.261955 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:22.262631 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:22.262759 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:22.266824 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:22.266831 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:22.266835 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:22.266875 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:22.270723 (+ 3848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:22.271563 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:22.271569 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:22.271572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:22.271767 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:22.272312 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:22.272317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:22.272320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:22.272752 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:22.272891 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:22.274578 (+ 1687us) server_negotiation.cc:299] Negotiation successful
0810 23:31:22.274837 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 23:31:22.279889 18851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 571a64b9bc554f4892b21227290e43b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31f9bdd654f047bb8c98a5e05e8bff7c. Latest consensus state: current_term: 1 leader_uuid: "31f9bdd654f047bb8c98a5e05e8bff7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f9bdd654f047bb8c98a5e05e8bff7c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37997 } } peers { permanent_uuid: "c2ebaaa659cc464282dcd46444f1024e" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 35865 } } peers { permanent_uuid: "571a64b9bc554f4892b21227290e43b5" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42283 } } }
I20260810 23:31:22.280248 18851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 571a64b9bc554f4892b21227290e43b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:31:22.281903 18695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2ebaaa659cc464282dcd46444f1024e: Bootstrap starting.
I20260810 23:31:22.283304 18881 mvcc.cc:205] Tried to move back new op lower bound from 7317113578529234944 to 7317113578382438400. Current Snapshot: MvccSnapshot[applied={T|T < 7317113578529234944}]
I20260810 23:31:22.284003 18835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31f9bdd654f047bb8c98a5e05e8bff7c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "31f9bdd654f047bb8c98a5e05e8bff7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f9bdd654f047bb8c98a5e05e8bff7c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37997 } } peers { permanent_uuid: "c2ebaaa659cc464282dcd46444f1024e" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 35865 } } peers { permanent_uuid: "571a64b9bc554f4892b21227290e43b5" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42283 } } }
I20260810 23:31:22.284359 18835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31f9bdd654f047bb8c98a5e05e8bff7c [sys.catalog]: This master's current role is: LEADER
I20260810 23:31:22.286003 18882 mvcc.cc:205] Tried to move back new op lower bound from 7317113578529234944 to 7317113578382438400. Current Snapshot: MvccSnapshot[applied={T|T < 7317113578529234944}]
I20260810 23:31:22.286158 18695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2ebaaa659cc464282dcd46444f1024e: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:22.287340 18851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 571a64b9bc554f4892b21227290e43b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "31f9bdd654f047bb8c98a5e05e8bff7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f9bdd654f047bb8c98a5e05e8bff7c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37997 } } peers { permanent_uuid: "c2ebaaa659cc464282dcd46444f1024e" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 35865 } } peers { permanent_uuid: "571a64b9bc554f4892b21227290e43b5" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42283 } } }
I20260810 23:31:22.287501 18695 log.cc:824] T 00000000000000000000000000000000 P c2ebaaa659cc464282dcd46444f1024e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:31:22.287585 18851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 571a64b9bc554f4892b21227290e43b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:31:22.287917 18873 catalog_manager.cc:1347] Generated new cluster ID: 7fe671fda4744c72a7212980f266a0cd
I20260810 23:31:22.288014 18873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:31:22.303918 18880 catalog_manager.cc:1259] Loaded cluster ID: 7fe671fda4744c72a7212980f266a0cd
I20260810 23:31:22.304039 18880 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 571a64b9bc554f4892b21227290e43b5: loading cluster ID for follower catalog manager: success
I20260810 23:31:22.308032 18695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2ebaaa659cc464282dcd46444f1024e: No bootstrap required, opened a new log
I20260810 23:31:22.309978 18873 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:31:22.312170 18873 catalog_manager.cc:1504] Loading token signing keys...
W20260810 23:31:22.312477 18880 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 571a64b9bc554f4892b21227290e43b5: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260810 23:31:22.312672 18880 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:31:22.319530 18695 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2ebaaa659cc464282dcd46444f1024e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f9bdd654f047bb8c98a5e05e8bff7c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37997 } } peers { permanent_uuid: "c2ebaaa659cc464282dcd46444f1024e" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 35865 } } peers { permanent_uuid: "571a64b9bc554f4892b21227290e43b5" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42283 } }
I20260810 23:31:22.307034 18707 master_runner.cc:186] Error getting master registration for 127.14.175.125:35865: OK, instance_id { permanent_uuid: "c2ebaaa659cc464282dcd46444f1024e" instance_seqno: 1786404681530222 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:31:22.327879 18707 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:31:22.327844 18695 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2ebaaa659cc464282dcd46444f1024e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ebaaa659cc464282dcd46444f1024e, State: Initialized, Role: FOLLOWER
I20260810 23:31:22.328679 18695 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2ebaaa659cc464282dcd46444f1024e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f9bdd654f047bb8c98a5e05e8bff7c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37997 } } peers { permanent_uuid: "c2ebaaa659cc464282dcd46444f1024e" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 35865 } } peers { permanent_uuid: "571a64b9bc554f4892b21227290e43b5" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42283 } }
I20260810 23:31:22.329445 18873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31f9bdd654f047bb8c98a5e05e8bff7c: Generated new TSK 0
I20260810 23:31:22.332913 18873 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:31:22.346231 18894 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2ebaaa659cc464282dcd46444f1024e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f9bdd654f047bb8c98a5e05e8bff7c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37997 } } peers { permanent_uuid: "c2ebaaa659cc464282dcd46444f1024e" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 35865 } } peers { permanent_uuid: "571a64b9bc554f4892b21227290e43b5" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42283 } } }
I20260810 23:31:22.346765 18894 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2ebaaa659cc464282dcd46444f1024e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:31:22.348853 18695 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2ebaaa659cc464282dcd46444f1024e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:31:22.348961 18654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31f9bdd654f047bb8c98a5e05e8bff7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c2ebaaa659cc464282dcd46444f1024e"
I20260810 23:31:22.351817 18653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2ebaaa659cc464282dcd46444f1024e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:22.365715 18653 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c2ebaaa659cc464282dcd46444f1024e [term 1 FOLLOWER]: Refusing update from remote peer 31f9bdd654f047bb8c98a5e05e8bff7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:31:22.368994 18654 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c2ebaaa659cc464282dcd46444f1024e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f9bdd654f047bb8c98a5e05e8bff7c in term 1.
I20260810 23:31:22.369294 18862 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 31f9bdd654f047bb8c98a5e05e8bff7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ebaaa659cc464282dcd46444f1024e" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 35865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:31:22.430832 18894 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2ebaaa659cc464282dcd46444f1024e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31f9bdd654f047bb8c98a5e05e8bff7c. Latest consensus state: current_term: 1 leader_uuid: "31f9bdd654f047bb8c98a5e05e8bff7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f9bdd654f047bb8c98a5e05e8bff7c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37997 } } peers { permanent_uuid: "c2ebaaa659cc464282dcd46444f1024e" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 35865 } } peers { permanent_uuid: "571a64b9bc554f4892b21227290e43b5" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42283 } } }
I20260810 23:31:22.431221 18894 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2ebaaa659cc464282dcd46444f1024e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:31:22.431597 18904 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2ebaaa659cc464282dcd46444f1024e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "31f9bdd654f047bb8c98a5e05e8bff7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f9bdd654f047bb8c98a5e05e8bff7c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37997 } } peers { permanent_uuid: "c2ebaaa659cc464282dcd46444f1024e" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 35865 } } peers { permanent_uuid: "571a64b9bc554f4892b21227290e43b5" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42283 } } }
I20260810 23:31:22.431998 18904 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2ebaaa659cc464282dcd46444f1024e [sys.catalog]: This master's current role is: FOLLOWER
W20260810 23:31:22.476585 15037 mini_kdc.cc:348] Time spent kdestroy: real 0.120s user 0.004s sys 0.029s
I20260810 23:31:22.503237 18927 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260810 23:31:22.516638 18829 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:31:22.506400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52902 (local address 127.14.175.124:42283)
0810 23:31:22.506802 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:22.506809 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:22.514684 (+ 7875us) server_negotiation.cc:407] Connection header received
0810 23:31:22.514967 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:22.514973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:22.515053 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:22.515120 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:22.516357 (+ 1237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52902: BlockingRecv error: recv got EOF from 127.0.0.1:52902 (error 108)
Metrics: {"server-negotiator.queue_time_us":247}
W20260810 23:31:22.518813 18831 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:31:22.512159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55134 (local address 127.14.175.126:37997)
0810 23:31:22.512388 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:22.512395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:22.517256 (+ 4861us) server_negotiation.cc:407] Connection header received
0810 23:31:22.517406 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:22.517412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:22.517500 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:22.517602 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:22.518559 (+ 957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55134: BlockingRecv error: recv got EOF from 127.0.0.1:55134 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260810 23:31:22.519505 18703 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:31:22.514736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40198 (local address 127.14.175.125:35865)
0810 23:31:22.514926 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:22.514932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:22.517915 (+ 2983us) server_negotiation.cc:407] Connection header received
0810 23:31:22.518152 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:22.518158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:22.518242 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:22.518308 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:22.519273 (+ 965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40198: BlockingRecv error: recv got EOF from 127.0.0.1:40198 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260810 23:31:22.551966 18703 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:31:22.541807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40202 (local address 127.14.175.125:35865)
0810 23:31:22.542017 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:22.542024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:22.549620 (+ 7596us) server_negotiation.cc:407] Connection header received
0810 23:31:22.550113 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:22.550124 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:22.550257 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:22.550335 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:22.551661 (+ 1326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40202: BlockingRecv error: recv got EOF from 127.0.0.1:40202 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260810 23:31:22.576385 18703 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:31:22.568109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40206 (local address 127.14.175.125:35865)
0810 23:31:22.568300 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:22.568306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:22.574711 (+ 6405us) server_negotiation.cc:407] Connection header received
0810 23:31:22.574965 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:22.574972 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:22.575090 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:22.575207 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:22.576146 (+ 939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40206: BlockingRecv error: recv got EOF from 127.0.0.1:40206 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260810 23:31:22.586030 18703 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:31:22.583470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40210 (local address 127.14.175.125:35865)
0810 23:31:22.583690 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:22.583697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:22.584376 (+ 679us) server_negotiation.cc:407] Connection header received
0810 23:31:22.584629 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:22.584635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:22.584748 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:22.584855 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:22.585770 (+ 915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40210: BlockingRecv error: recv got EOF from 127.0.0.1:40210 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260810 23:31:22.597500 18703 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:31:22.595155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40212 (local address 127.14.175.125:35865)
0810 23:31:22.595341 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:22.595347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:22.595916 (+ 569us) server_negotiation.cc:407] Connection header received
0810 23:31:22.596115 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:22.596120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:22.596215 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:22.596325 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:22.597257 (+ 932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40212: BlockingRecv error: recv got EOF from 127.0.0.1:40212 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260810 23:31:22.611209 18703 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:31:22.608943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40222 (local address 127.14.175.125:35865)
0810 23:31:22.609181 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:22.609188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:22.609731 (+ 543us) server_negotiation.cc:407] Connection header received
0810 23:31:22.609945 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:22.609951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:22.610048 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:22.610138 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:22.610975 (+ 837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40222: BlockingRecv error: recv got EOF from 127.0.0.1:40222 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260810 23:31:22.633692 18703 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:31:22.631182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40232 (local address 127.14.175.125:35865)
0810 23:31:22.631384 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:22.631390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:22.632025 (+ 635us) server_negotiation.cc:407] Connection header received
0810 23:31:22.632234 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:22.632240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:22.632344 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:22.632464 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:22.633433 (+ 969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40232: BlockingRecv error: recv got EOF from 127.0.0.1:40232 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260810 23:31:22.672497 18703 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:31:22.669762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40242 (local address 127.14.175.125:35865)
0810 23:31:22.669964 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:22.669972 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:22.670572 (+ 600us) server_negotiation.cc:407] Connection header received
0810 23:31:22.670928 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:22.670935 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:22.671052 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:22.671177 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:22.672205 (+ 1028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40242: BlockingRecv error: recv got EOF from 127.0.0.1:40242 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260810 23:31:22.745404 18703 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:31:22.742839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40248 (local address 127.14.175.125:35865)
0810 23:31:22.743086 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:22.743093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:22.743716 (+ 623us) server_negotiation.cc:407] Connection header received
0810 23:31:22.743932 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:22.743938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:22.744046 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:22.744154 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:22.745133 (+ 979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40248: BlockingRecv error: recv got EOF from 127.0.0.1:40248 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 23:31:22.757047 18862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31f9bdd654f047bb8c98a5e05e8bff7c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "31f9bdd654f047bb8c98a5e05e8bff7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f9bdd654f047bb8c98a5e05e8bff7c" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 37997 } } peers { permanent_uuid: "c2ebaaa659cc464282dcd46444f1024e" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 35865 } } peers { permanent_uuid: "571a64b9bc554f4892b21227290e43b5" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 42283 } } }
I20260810 23:31:22.757421 18862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31f9bdd654f047bb8c98a5e05e8bff7c [sys.catalog]: This master's current role is: LEADER
W20260810 23:31:22.878968 18703 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:31:22.876834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40250 (local address 127.14.175.125:35865)
0810 23:31:22.877028 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:22.877033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:22.877458 (+ 425us) server_negotiation.cc:407] Connection header received
0810 23:31:22.877661 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:22.877665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:22.877759 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:22.877850 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:22.878687 (+ 837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40250: BlockingRecv error: recv got EOF from 127.0.0.1:40250 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260810 23:31:23.144802 18703 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:31:23.142908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40264 (local address 127.14.175.125:35865)
0810 23:31:23.143166 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:23.143173 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:23.143298 (+ 125us) server_negotiation.cc:407] Connection header received
0810 23:31:23.143535 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:23.143541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:23.143670 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:23.143773 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:23.144579 (+ 806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40264: BlockingRecv error: recv got EOF from 127.0.0.1:40264 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
I20260810 23:31:23.316197 18880 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 571a64b9bc554f4892b21227290e43b5: acquiring CA information for follower catalog manager: success
I20260810 23:31:23.318339 18880 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 571a64b9bc554f4892b21227290e43b5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:31:23.508616 18927 catalog_manager.cc:1259] Loaded cluster ID: 7fe671fda4744c72a7212980f266a0cd
I20260810 23:31:23.508749 18927 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c2ebaaa659cc464282dcd46444f1024e: loading cluster ID for follower catalog manager: success
I20260810 23:31:23.511540 18927 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c2ebaaa659cc464282dcd46444f1024e: acquiring CA information for follower catalog manager: success
I20260810 23:31:23.513767 18927 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c2ebaaa659cc464282dcd46444f1024e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:31:23.670689 19029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:23.663082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40268 (local address 127.14.175.125:35865)
0810 23:31:23.663775 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:23.663781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:23.663955 (+ 174us) server_negotiation.cc:407] Connection header received
0810 23:31:23.664203 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:23.664207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:23.664380 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:23.664510 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:31:23.665026 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:23.665984 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:23.668058 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:23.668565 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:23.668671 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:23.669391 (+ 720us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:31:23.670023 (+ 632us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:31:23.670143 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:23.670293 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 23:31:23.670372 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":327,"threads_started":1}
I20260810 23:31:23.686604 19035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:23.679570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52918 (local address 127.14.175.124:42283)
0810 23:31:23.680082 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:23.680089 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:23.680452 (+ 363us) server_negotiation.cc:407] Connection header received
0810 23:31:23.680703 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:23.680708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:23.680884 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:23.681034 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:31:23.681525 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:23.682265 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:23.684224 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:23.684653 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:23.684753 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:23.685370 (+ 617us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:31:23.685945 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:31:23.686055 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:23.686210 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 23:31:23.686288 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":239,"threads_started":1}
I20260810 23:31:23.713567 19035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:23.695198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52928 (local address 127.14.175.124:42283)
0810 23:31:23.695383 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:23.695391 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:23.706557 (+ 11166us) server_negotiation.cc:407] Connection header received
0810 23:31:23.706952 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:23.706959 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:23.707070 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:23.707135 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:31:23.707928 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:23.709030 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:23.711287 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:23.711905 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:23.712009 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:23.712681 (+ 672us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:31:23.712822 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:31:23.712887 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:23.713132 (+ 245us) server_negotiation.cc:299] Negotiation successful
0810 23:31:23.713248 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 23:31:23.715363 19047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:23.694709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55150 (local address 127.14.175.126:37997)
0810 23:31:23.705563 (+ 10854us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:23.705571 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:23.706907 (+ 1336us) server_negotiation.cc:407] Connection header received
0810 23:31:23.707045 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:23.707050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:23.707294 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:23.707479 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:31:23.708372 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:23.709615 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:23.712004 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:23.712707 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:23.712825 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:23.713561 (+ 736us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:31:23.714054 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:31:23.714163 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:23.714971 (+ 808us) server_negotiation.cc:299] Negotiation successful
0810 23:31:23.715063 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10647,"thread_start_us":10876,"threads_started":1}
I20260810 23:31:23.720755 19029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:23.706849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40276 (local address 127.14.175.125:35865)
0810 23:31:23.707045 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:23.707050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:23.714797 (+ 7747us) server_negotiation.cc:407] Connection header received
0810 23:31:23.714950 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:23.714960 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:23.715048 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:23.715136 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:31:23.715688 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:23.716815 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:23.718774 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:23.719393 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:23.719515 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:23.719935 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:31:23.720080 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:31:23.720183 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:23.720359 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 23:31:23.720455 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 23:31:23.721239 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 18459
I20260810 23:31:23.722386 18563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:31:23.862442 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 18459
I20260810 23:31:23.890367 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 18584
I20260810 23:31:23.891618 18690 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:31:24.002043 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 18584
I20260810 23:31:24.031203 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 18707
I20260810 23:31:24.032322 18815 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:31:24.139464 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 18707
2026-08-10T23:31:24Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5254 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:31:24 dist-test-slave-rmhg krb5kdc[19056](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:31:24 dist-test-slave-rmhg krb5kdc[19056](info): set up 2 sockets
Aug 10 23:31:24 dist-test-slave-rmhg krb5kdc[19056](info): commencing operation
krb5kdc: starting...
W20260810 23:31:25.438982 15037 mini_kdc.cc:121] Time spent starting KDC: real 1.250s user 0.007s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/krb5kdc/test-user.keytab.
Aug 10 23:31:25 dist-test-slave-rmhg krb5kdc[19056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:31:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:31:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.175.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
Entry for principal kudu/127.14.175.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab.
I20260810 23:31:25.792950 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:34059
--webserver_interface=127.14.175.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:41189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:34059,127.14.175.125:36277,127.14.175.124:38045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:31:26.146418 19077 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:31:26.146791 19077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:31:26.146895 19077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:31:26.147013 19077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:31:26.154613 19077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:31:26.154734 19077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:31:26.154819 19077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:31:26.154897 19077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:31:26.154984 19077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:31:26.165360 19077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:41189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:34059,127.14.175.125:36277,127.14.175.124:38045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.126.keytab
--principal=kudu/127.14.175.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.126:34059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.175.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:31:26.166759 19077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:31:26.168164 19077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:31:26.180615 19083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:26.180721 19087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:26.180699 19085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:31:26.181638 19077 server_base.cc:1034] running on GCE node
I20260810 23:31:26.182512 19077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:31:26.184062 19077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:31:26.185281 19077 hybrid_clock.cc:648] HybridClock initialized: now 1786404686185203 us; error 45 us; skew 500 ppm
I20260810 23:31:26.185652 19077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:31:26 dist-test-slave-rmhg krb5kdc[19056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404686, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:31:26.193737 19077 init.cc:377] Logged in from keytab as kudu/127.14.175.126@KRBTEST.COM (short username kudu)
I20260810 23:31:26.195513 19077 webserver.cc:460] Webserver started at http://127.14.175.126:44785/ using document root <none> and password file <none>
I20260810 23:31:26.196260 19077 fs_manager.cc:359] Metadata directory not provided
I20260810 23:31:26.196357 19077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:31:26.196686 19077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:31:26.199970 19077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-0/data/instance:
uuid: "b773c2118c9e4c15b459f0e53fb00713"
format_stamp: "Formatted at 2026-08-10 23:31:26 on dist-test-slave-rmhg"
server_key: "8685613be0c335afc8a9177ae5fa07ef"
server_key_iv: "24c00f6100e8e5818c6ebf69f2066375"
server_key_version: "encryptionkey@0"
I20260810 23:31:26.200829 19077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-0/wal/instance:
uuid: "b773c2118c9e4c15b459f0e53fb00713"
format_stamp: "Formatted at 2026-08-10 23:31:26 on dist-test-slave-rmhg"
server_key: "8685613be0c335afc8a9177ae5fa07ef"
server_key_iv: "24c00f6100e8e5818c6ebf69f2066375"
server_key_version: "encryptionkey@0"
I20260810 23:31:26.206357 19077 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 23:31:26.210026 19097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:26.211407 19077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:31:26.211593 19077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-0/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-0/wal
uuid: "b773c2118c9e4c15b459f0e53fb00713"
format_stamp: "Formatted at 2026-08-10 23:31:26 on dist-test-slave-rmhg"
server_key: "8685613be0c335afc8a9177ae5fa07ef"
server_key_iv: "24c00f6100e8e5818c6ebf69f2066375"
server_key_version: "encryptionkey@0"
I20260810 23:31:26.211824 19077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:31:26.223798 19077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:31:26.228298 19077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:31:26.228569 19077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:31:26.247480 19077 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.126:34059
I20260810 23:31:26.247510 19183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.126:34059 every 8 connection(s)
I20260810 23:31:26.248929 19077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-0/data/info.pb
I20260810 23:31:26.252913 19185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:26.256963 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 19077
I20260810 23:31:26.257234 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-0/wal/instance
I20260810 23:31:26.257822 15037 external_mini_cluster.cc:1428] Setting key acaf4b11cae91f85e2833d50cfd02dc5
I20260810 23:31:26.261394 19185 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34059 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:31:26 dist-test-slave-rmhg krb5kdc[19056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404686, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:31:26.285868 19189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:26.265434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43030 (local address 127.14.175.126:34059)
0810 23:31:26.267214 (+ 1780us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:26.267226 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:26.267282 (+ 56us) server_negotiation.cc:407] Connection header received
0810 23:31:26.268911 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:26.268917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:26.269223 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:26.269428 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:26.270807 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:26.272052 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:26.273510 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:26.274091 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:26.274271 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:26.278184 (+ 3913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:26.278193 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:26.278207 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:26.278244 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:26.281089 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:26.281939 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:26.281946 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:26.281948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:26.282121 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:26.282538 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:26.282541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:26.282543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:26.283114 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:26.283323 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:26.283916 (+ 593us) server_negotiation.cc:299] Negotiation successful
0810 23:31:26.284249 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1555,"thread_start_us":1330,"threads_started":1}
I20260810 23:31:26.285897 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:26.265434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:34059 (local address 127.0.0.1:43030)
0810 23:31:26.266360 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:26.266421 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:26.268507 (+ 2086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:26.269456 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:26.269463 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:26.270218 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:26.270592 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:26.270613 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:26.272201 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:26.272205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:26.273374 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:26.273388 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:26.274335 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:26.274343 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:26.274524 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:26.275226 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:26.275254 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:26.277914 (+ 2660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:26.281288 (+ 3374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:26.281304 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:26.281320 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:26.281782 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:26.282243 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:26.282249 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:26.282253 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:26.282423 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:26.283267 (+ 844us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:26.283276 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:26.283500 (+ 224us) client_negotiation.cc:769] Sending connection context
0810 23:31:26.283740 (+ 240us) client_negotiation.cc:240] Negotiation successful
0810 23:31:26.284399 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":155,"thread_start_us":387,"threads_started":1}
I20260810 23:31:26.289554 19185 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.125" port: 36277 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:31:26.290594 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:26.290172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:36277 (local address 127.0.0.1:59384)
0810 23:31:26.290356 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:26.290468 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.125:36277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
WARNING: no policy specified for kudu/127.14.175.125@KRBTEST.COM; defaulting to no policy
W20260810 23:31:26.290846 19102 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.175.125:36277: connect: Connection refused (error 111)
I20260810 23:31:26.292438 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:26.292037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:36277 (local address 127.0.0.1:59398)
0810 23:31:26.292219 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:26.292292 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.125:36277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 23:31:26.292647 19185 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.125:36277: Network error: Client connection negotiation failed: client connection to 127.14.175.125:36277: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.125.keytab.
Entry for principal kudu/127.14.175.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.125.keytab.
I20260810 23:31:26.333060 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.125:36277
--webserver_interface=127.14.175.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.125.keytab
--principal=kudu/127.14.175.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:41189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:34059,127.14.175.125:36277,127.14.175.124:38045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:31:26.342034 19185 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.125" port: 36277 } attempt: 1
I20260810 23:31:26.343410 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:26.342859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:36277 (local address 127.0.0.1:59408)
0810 23:31:26.343087 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:26.343246 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.125:36277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260810 23:31:26.344558 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:26.344121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:36277 (local address 127.0.0.1:59420)
0810 23:31:26.344271 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:26.344364 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.125:36277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 23:31:26.344846 19185 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.125:36277: Network error: Client connection negotiation failed: client connection to 127.14.175.125:36277: connect: Connection refused (error 111)
I20260810 23:31:26.413130 19185 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.125" port: 36277 } attempt: 2
I20260810 23:31:26.414414 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:26.413922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:36277 (local address 127.0.0.1:59432)
0810 23:31:26.414170 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:26.414266 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.125:36277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260810 23:31:26.415442 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:26.415070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:36277 (local address 127.0.0.1:59444)
0810 23:31:26.415231 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:26.415303 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.125:36277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 23:31:26.415668 19185 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.125:36277: Network error: Client connection negotiation failed: client connection to 127.14.175.125:36277: connect: Connection refused (error 111)
I20260810 23:31:26.506942 19185 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.125" port: 36277 } attempt: 3
I20260810 23:31:26.508250 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:26.507789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:36277 (local address 127.0.0.1:59454)
0810 23:31:26.508005 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:26.508101 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.125:36277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260810 23:31:26.509325 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:26.508912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:36277 (local address 127.0.0.1:59470)
0810 23:31:26.509056 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:26.509129 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.125:36277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 23:31:26.509518 19185 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.125:36277: Network error: Client connection negotiation failed: client connection to 127.14.175.125:36277: connect: Connection refused (error 111)
I20260810 23:31:26.652781 19185 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.125" port: 36277 } attempt: 4
I20260810 23:31:26.654137 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:26.653556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:36277 (local address 127.0.0.1:59484)
0810 23:31:26.653905 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:26.653996 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.125:36277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198}
I20260810 23:31:26.655166 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:26.654796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:36277 (local address 127.0.0.1:59496)
0810 23:31:26.654946 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:26.655027 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.125:36277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 23:31:26.655368 19185 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.125:36277: Network error: Client connection negotiation failed: client connection to 127.14.175.125:36277: connect: Connection refused (error 111)
W20260810 23:31:26.691051 19194 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:31:26.691399 19194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:31:26.691510 19194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:31:26.691633 19194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:31:26.699117 19194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:31:26.699244 19194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:31:26.699322 19194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:31:26.699396 19194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:31:26.699483 19194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:31:26.709995 19194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:41189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:34059,127.14.175.125:36277,127.14.175.124:38045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.125.keytab
--principal=kudu/127.14.175.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.125:36277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.175.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:31:26.711367 19194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:31:26.712764 19194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:31:26.724992 19206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:26.725040 19203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:26.725090 19204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:31:26.731045 19194 server_base.cc:1034] running on GCE node
I20260810 23:31:26.732002 19194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:31:26.733671 19194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:31:26.734917 19194 hybrid_clock.cc:648] HybridClock initialized: now 1786404686734833 us; error 62 us; skew 500 ppm
I20260810 23:31:26.735294 19194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:31:26 dist-test-slave-rmhg krb5kdc[19056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404686, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:31:26.738679 19194 init.cc:377] Logged in from keytab as kudu/127.14.175.125@KRBTEST.COM (short username kudu)
I20260810 23:31:26.740352 19194 webserver.cc:460] Webserver started at http://127.14.175.125:37923/ using document root <none> and password file <none>
I20260810 23:31:26.741145 19194 fs_manager.cc:359] Metadata directory not provided
I20260810 23:31:26.741315 19194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:31:26.741662 19194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:31:26.744917 19194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-1/data/instance:
uuid: "5860c14d4d9b4aab9ba7c593cd24990a"
format_stamp: "Formatted at 2026-08-10 23:31:26 on dist-test-slave-rmhg"
server_key: "922954d383a40e1297c1fcdb617c2ceb"
server_key_iv: "23d7d4b18d7e34f5e04afb31c1a1acc8"
server_key_version: "encryptionkey@0"
I20260810 23:31:26.745839 19194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-1/wal/instance:
uuid: "5860c14d4d9b4aab9ba7c593cd24990a"
format_stamp: "Formatted at 2026-08-10 23:31:26 on dist-test-slave-rmhg"
server_key: "922954d383a40e1297c1fcdb617c2ceb"
server_key_iv: "23d7d4b18d7e34f5e04afb31c1a1acc8"
server_key_version: "encryptionkey@0"
I20260810 23:31:26.751478 19194 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260810 23:31:26.755107 19216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:26.756517 19194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:31:26.756691 19194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-1/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-1/wal
uuid: "5860c14d4d9b4aab9ba7c593cd24990a"
format_stamp: "Formatted at 2026-08-10 23:31:26 on dist-test-slave-rmhg"
server_key: "922954d383a40e1297c1fcdb617c2ceb"
server_key_iv: "23d7d4b18d7e34f5e04afb31c1a1acc8"
server_key_version: "encryptionkey@0"
I20260810 23:31:26.756908 19194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:31:26.784291 19194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:31:26.788607 19194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:31:26.788888 19194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:31:26.807179 19194 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.125:36277
I20260810 23:31:26.807217 19287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.125:36277 every 8 connection(s)
I20260810 23:31:26.808672 19194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-1/data/info.pb
I20260810 23:31:26.812606 19288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:26.818706 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 19194
I20260810 23:31:26.818902 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-1/wal/instance
I20260810 23:31:26.819324 15037 external_mini_cluster.cc:1428] Setting key b8037ef9a98e2438bdebd6f14b5606c1
I20260810 23:31:26.819607 19288 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34059 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:31:26 dist-test-slave-rmhg krb5kdc[19056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404686, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.125@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:31:26.841260 19290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:26.822512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43042 (local address 127.14.175.126:34059)
0810 23:31:26.823119 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:26.823124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:26.824334 (+ 1210us) server_negotiation.cc:407] Connection header received
0810 23:31:26.826443 (+ 2109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:26.826449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:26.826537 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:26.826635 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:26.828424 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:26.829278 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:26.830921 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:26.831366 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:26.831485 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:26.835169 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:26.835182 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:26.835185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:26.835229 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:26.837994 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:26.838835 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:26.838840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:26.838843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:26.838970 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:26.839406 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:26.839409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:26.839412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:26.839717 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:26.839854 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:26.840603 (+ 749us) server_negotiation.cc:299] Negotiation successful
0810 23:31:26.840838 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":346,"threads_started":1}
I20260810 23:31:26.842309 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:26.823299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:34059 (local address 127.0.0.1:43042)
0810 23:31:26.824147 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:26.824194 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:26.826111 (+ 1917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:26.826944 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:26.826954 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:26.827558 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:26.828182 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:26.828201 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:26.829449 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:26.829453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:26.830753 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:26.830767 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:26.831633 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:26.831642 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:26.831906 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:26.832592 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:26.832622 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:26.834912 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:26.838167 (+ 3255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:26.838178 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:26.838197 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:26.838704 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:26.839084 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:26.839089 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:26.839093 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:26.839293 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:26.839850 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:26.839859 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:26.840242 (+ 383us) client_negotiation.cc:769] Sending connection context
0810 23:31:26.840502 (+ 260us) client_negotiation.cc:240] Negotiation successful
0810 23:31:26.841070 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":354,"threads_started":1}
I20260810 23:31:26.844255 19288 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.125" port: 36277 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.175.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.175.124@KRBTEST.COM" created.
Aug 10 23:31:26 dist-test-slave-rmhg krb5kdc[19056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404686, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.125@KRBTEST.COM for kudu/127.14.175.125@KRBTEST.COM
I20260810 23:31:26.862107 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:26.844978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:36277 (local address 127.0.0.1:59504)
0810 23:31:26.845251 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:26.845275 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:26.845367 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:26.846479 (+ 1112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:26.846483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:26.846918 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:26.847057 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:26.847065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:26.848419 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:26.848424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:26.849316 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:26.849329 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:26.850119 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:26.850124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:26.850314 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:26.850931 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:26.850948 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:26.855644 (+ 4696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:26.859515 (+ 3871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:26.859519 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:26.859522 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:26.859920 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:26.860338 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:26.860342 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:26.860344 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:26.860488 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:26.861302 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:26.861307 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:26.861447 (+ 140us) client_negotiation.cc:769] Sending connection context
0810 23:31:26.861556 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 23:31:26.861839 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260810 23:31:26.862406 19293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:26.845370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59504 (local address 127.14.175.125:36277)
0810 23:31:26.846058 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:26.846064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:26.846091 (+ 27us) server_negotiation.cc:407] Connection header received
0810 23:31:26.846134 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:26.846140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:26.846330 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:26.846464 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:26.847220 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:26.848288 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:26.849439 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:26.849982 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:26.850093 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:26.855893 (+ 5800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:26.855910 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:26.855930 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:26.855973 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:26.859352 (+ 3379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:26.860065 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:26.860074 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:26.860077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:26.860227 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:26.860594 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:26.860598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:26.860601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:26.861060 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:26.861352 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:26.861753 (+ 401us) server_negotiation.cc:299] Negotiation successful
0810 23:31:26.862003 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":397,"threads_started":1}
I20260810 23:31:26.864684 19288 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.124" port: 38045 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:31:26.865893 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:26.865388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:38045 (local address 127.0.0.1:47274)
0810 23:31:26.865559 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:26.865720 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260810 23:31:26.866149 19219 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.175.124:38045: connect: Connection refused (error 111)
I20260810 23:31:26.867758 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:26.867358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:38045 (local address 127.0.0.1:47280)
0810 23:31:26.867537 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:26.867633 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 23:31:26.867985 19288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.124:38045: Network error: Client connection negotiation failed: client connection to 127.14.175.124:38045: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.175.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.124.keytab.
Entry for principal kudu/127.14.175.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.124.keytab.
I20260810 23:31:26.892359 15037 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
/tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.124:38045
--webserver_interface=127.14.175.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.124.keytab
--principal=kudu/127.14.175.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.175.84:41189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.175.126:34059,127.14.175.125:36277,127.14.175.124:38045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:31:26.917310 19288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.124" port: 38045 } attempt: 1
I20260810 23:31:26.918450 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:26.917983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:38045 (local address 127.0.0.1:47284)
0810 23:31:26.918194 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:26.918296 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 23:31:26.919718 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:26.919246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:38045 (local address 127.0.0.1:47286)
0810 23:31:26.919453 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:26.919560 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260810 23:31:26.919948 19288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.124:38045: Network error: Client connection negotiation failed: client connection to 127.14.175.124:38045: connect: Connection refused (error 111)
I20260810 23:31:26.954615 19185 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.125" port: 36277 } attempt: 5
Aug 10 23:31:26 dist-test-slave-rmhg krb5kdc[19056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404686, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for kudu/127.14.175.125@KRBTEST.COM
I20260810 23:31:26.968446 19293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:26.955477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59512 (local address 127.14.175.125:36277)
0810 23:31:26.955662 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:26.955667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:26.955691 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:31:26.955858 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:26.955864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:26.955951 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:26.956040 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:26.956877 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:26.957647 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:26.958851 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:26.959305 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:26.959421 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:26.963101 (+ 3680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:26.963106 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:26.963110 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:26.963144 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:26.965754 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:26.966600 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:26.966605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:26.966607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:26.966736 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:26.967147 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:26.967151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:26.967153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:26.967446 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:26.967559 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:26.967990 (+ 431us) server_negotiation.cc:299] Negotiation successful
0810 23:31:26.968170 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:31:26.968662 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:26.955350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:36277 (local address 127.0.0.1:59512)
0810 23:31:26.955548 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:26.955577 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:26.955723 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:26.956102 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:26.956108 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:26.956526 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:26.956718 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:26.956728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:26.957792 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:26.957798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:26.958691 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:26.958705 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:26.959457 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:26.959464 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:26.959690 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:26.960336 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:26.960354 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:26.962886 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:26.965915 (+ 3029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:26.965921 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:26.965925 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:26.966482 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:26.966845 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:26.966850 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:26.966853 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:26.967035 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:26.967560 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:26.967565 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:26.967760 (+ 195us) client_negotiation.cc:769] Sending connection context
0810 23:31:26.967921 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 23:31:26.968298 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 23:31:26.969744 19185 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.124" port: 38045 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:31:26.971009 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:26.970535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:38045 (local address 127.0.0.1:47292)
0810 23:31:26.970728 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:26.970834 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 23:31:26.971997 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:26.971598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:38045 (local address 127.0.0.1:47308)
0810 23:31:26.971772 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:26.971866 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 23:31:26.972235 19185 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.124:38045: Network error: Client connection negotiation failed: client connection to 127.14.175.124:38045: connect: Connection refused (error 111)
I20260810 23:31:26.988190 19288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.124" port: 38045 } attempt: 2
I20260810 23:31:26.989416 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:26.988913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:38045 (local address 127.0.0.1:47310)
0810 23:31:26.989117 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:26.989253 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260810 23:31:26.990399 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:26.990082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:38045 (local address 127.0.0.1:47320)
0810 23:31:26.990217 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:26.990286 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 23:31:26.990599 19288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.124:38045: Network error: Client connection negotiation failed: client connection to 127.14.175.124:38045: connect: Connection refused (error 111)
I20260810 23:31:27.023456 19185 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.124" port: 38045 } attempt: 1
I20260810 23:31:27.024659 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:27.024184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:38045 (local address 127.0.0.1:47322)
0810 23:31:27.024380 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:27.024489 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 23:31:27.025653 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:27.025337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:38045 (local address 127.0.0.1:47332)
0810 23:31:27.025481 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:27.025551 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 23:31:27.025866 19185 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.124:38045: Network error: Client connection negotiation failed: client connection to 127.14.175.124:38045: connect: Connection refused (error 111)
I20260810 23:31:27.081812 19288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.124" port: 38045 } attempt: 3
I20260810 23:31:27.083042 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:27.082579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:38045 (local address 127.0.0.1:47344)
0810 23:31:27.082780 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:27.082876 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260810 23:31:27.084113 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:27.083681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:38045 (local address 127.0.0.1:47356)
0810 23:31:27.083908 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:27.083990 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260810 23:31:27.084332 19288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.124:38045: Network error: Client connection negotiation failed: client connection to 127.14.175.124:38045: connect: Connection refused (error 111)
I20260810 23:31:27.094048 19185 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.124" port: 38045 } attempt: 2
I20260810 23:31:27.095109 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:27.094641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:38045 (local address 127.0.0.1:47358)
0810 23:31:27.094865 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:27.094948 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 23:31:27.096099 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:27.095788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:38045 (local address 127.0.0.1:47374)
0810 23:31:27.095930 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:27.095996 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 23:31:27.096306 19185 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.124:38045: Network error: Client connection negotiation failed: client connection to 127.14.175.124:38045: connect: Connection refused (error 111)
I20260810 23:31:27.202526 19185 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.124" port: 38045 } attempt: 3
I20260810 23:31:27.203754 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:27.203239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:38045 (local address 127.0.0.1:47376)
0810 23:31:27.203464 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:27.203566 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260810 23:31:27.204876 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:27.204495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:38045 (local address 127.0.0.1:47380)
0810 23:31:27.204692 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:27.204762 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260810 23:31:27.205083 19185 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.124:38045: Network error: Client connection negotiation failed: client connection to 127.14.175.124:38045: connect: Connection refused (error 111)
I20260810 23:31:27.227532 19288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.124" port: 38045 } attempt: 4
I20260810 23:31:27.228585 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:27.228109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:38045 (local address 127.0.0.1:47386)
0810 23:31:27.228339 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:27.228423 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260810 23:31:27.229560 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:27.229244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:38045 (local address 127.0.0.1:47392)
0810 23:31:27.229383 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:27.229453 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.175.124:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 23:31:27.229768 19288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.175.124:38045: Network error: Client connection negotiation failed: client connection to 127.14.175.124:38045: connect: Connection refused (error 111)
W20260810 23:31:27.242892 19299 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:31:27.243266 19299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:31:27.243368 19299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:31:27.243484 19299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:31:27.250972 19299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:31:27.251086 19299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:31:27.251169 19299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:31:27.251236 19299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:31:27.251324 19299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:31:27.261586 19299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.175.84:41189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.175.126:34059,127.14.175.125:36277,127.14.175.124:38045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/krb5kdc/kudu_127.14.175.124.keytab
--principal=kudu/127.14.175.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.175.124:38045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.175.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:31:27.262979 19299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:31:27.264359 19299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:31:27.276211 19306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:27.276254 19307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:27.276281 19311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:31:27.276842 19299 server_base.cc:1034] running on GCE node
I20260810 23:31:27.277791 19299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:31:27.279223 19299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:31:27.280431 19299 hybrid_clock.cc:648] HybridClock initialized: now 1786404687280353 us; error 49 us; skew 500 ppm
I20260810 23:31:27.280762 19299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:31:27 dist-test-slave-rmhg krb5kdc[19056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404687, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:31:27.284083 19299 init.cc:377] Logged in from keytab as kudu/127.14.175.124@KRBTEST.COM (short username kudu)
I20260810 23:31:27.285729 19299 webserver.cc:460] Webserver started at http://127.14.175.124:42507/ using document root <none> and password file <none>
I20260810 23:31:27.286463 19299 fs_manager.cc:359] Metadata directory not provided
I20260810 23:31:27.286561 19299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:31:27.286849 19299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:31:27.290189 19299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-2/data/instance:
uuid: "2186c103cb5e4549923d4eb7b21340c1"
format_stamp: "Formatted at 2026-08-10 23:31:27 on dist-test-slave-rmhg"
server_key: "279d461652673a40ca9ba2b41149aa8d"
server_key_iv: "3a107efced9a5befbec2de2a03d45485"
server_key_version: "encryptionkey@0"
I20260810 23:31:27.291078 19299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-2/wal/instance:
uuid: "2186c103cb5e4549923d4eb7b21340c1"
format_stamp: "Formatted at 2026-08-10 23:31:27 on dist-test-slave-rmhg"
server_key: "279d461652673a40ca9ba2b41149aa8d"
server_key_iv: "3a107efced9a5befbec2de2a03d45485"
server_key_version: "encryptionkey@0"
I20260810 23:31:27.296981 19299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:31:27.300916 19325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:27.302541 19299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:31:27.302747 19299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-2/data,/tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-2/wal
uuid: "2186c103cb5e4549923d4eb7b21340c1"
format_stamp: "Formatted at 2026-08-10 23:31:27 on dist-test-slave-rmhg"
server_key: "279d461652673a40ca9ba2b41149aa8d"
server_key_iv: "3a107efced9a5befbec2de2a03d45485"
server_key_version: "encryptionkey@0"
I20260810 23:31:27.302990 19299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:31:27.330960 19299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:31:27.335088 19299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:31:27.335350 19299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:31:27.353711 19299 rpc_server.cc:307] RPC server started. Bound to: 127.14.175.124:38045
I20260810 23:31:27.353750 19398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.175.124:38045 every 8 connection(s)
I20260810 23:31:27.355147 19299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-2/data/info.pb
I20260810 23:31:27.358862 19400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:27.365381 15037 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu as pid 19299
I20260810 23:31:27.365568 15037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwnDIbQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404299514076-15037-0/minicluster-data/master-2/wal/instance
I20260810 23:31:27.366029 15037 external_mini_cluster.cc:1428] Setting key 0db76c3c784d106ae0b1889e3b6380a7
I20260810 23:31:27.365614 19400 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34059 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:31:27 dist-test-slave-rmhg krb5kdc[19056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404687, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.124@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
Aug 10 23:31:27 dist-test-slave-rmhg krb5kdc[19056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.124@KRBTEST.COM
I20260810 23:31:27.382323 19185 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.124" port: 38045 } attempt: 4
I20260810 23:31:27.388242 19407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:27.368535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43054 (local address 127.14.175.126:34059)
0810 23:31:27.369059 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:27.369065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:27.369689 (+ 624us) server_negotiation.cc:407] Connection header received
0810 23:31:27.370868 (+ 1179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:27.370874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:27.370969 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:27.371087 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:27.372850 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:27.373966 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:27.376039 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:27.376640 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:27.376809 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:27.380746 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:27.380751 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:27.380755 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:27.380792 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:27.384194 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:27.385397 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:27.385404 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:27.385408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:27.385562 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:27.386143 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:27.386149 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:27.386151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:27.386520 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:27.386650 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:27.387565 (+ 915us) server_negotiation.cc:299] Negotiation successful
0810 23:31:27.387817 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":253,"threads_started":1}
I20260810 23:31:27.390074 19406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:27.367811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47394 (local address 127.14.175.124:38045)
0810 23:31:27.368587 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:27.368597 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:27.368638 (+ 41us) server_negotiation.cc:407] Connection header received
0810 23:31:27.370600 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:27.370624 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:27.370912 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:27.371138 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:27.372954 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:27.374516 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:27.376292 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:27.377072 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:27.377207 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:27.381420 (+ 4213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:27.381430 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:27.381444 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:27.381479 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:27.385456 (+ 3977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:27.386368 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:27.386377 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:27.386379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:27.386532 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:27.387028 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:27.387033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:27.387036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:27.387512 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:27.387823 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:27.388597 (+ 774us) server_negotiation.cc:299] Negotiation successful
0810 23:31:27.388848 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":332,"threads_started":1}
I20260810 23:31:27.390106 19408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:27.368620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.126:34059 (local address 127.0.0.1:43054)
0810 23:31:27.369501 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:27.369565 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:27.370609 (+ 1044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:27.371171 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:27.371179 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:27.372070 (+ 891us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:27.372645 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:27.372676 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:27.374141 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:27.374145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:27.375859 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:27.375876 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:27.376883 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:27.376888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:27.377072 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:27.377852 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:27.377889 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:27.380533 (+ 2644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:27.384344 (+ 3811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:27.384359 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:27.384378 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:27.385274 (+ 896us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:27.385723 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:27.385727 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:27.385731 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:27.386001 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:27.386656 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:27.386665 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:27.387078 (+ 413us) client_negotiation.cc:769] Sending connection context
0810 23:31:27.387421 (+ 343us) client_negotiation.cc:240] Negotiation successful
0810 23:31:27.387991 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":413,"threads_started":1}
Aug 10 23:31:27 dist-test-slave-rmhg krb5kdc[19056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404686, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.126@KRBTEST.COM for kudu/127.14.175.124@KRBTEST.COM
I20260810 23:31:27.392904 19400 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.125" port: 36277 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:31:27.399873 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:27.382970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:38045 (local address 127.0.0.1:47406)
0810 23:31:27.383236 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:27.383261 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:27.383376 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:27.385708 (+ 2332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:27.385714 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:27.386127 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:27.386344 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:27.386355 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:27.387871 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:27.387876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:27.389091 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:27.389112 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:27.390148 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:27.390154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:27.390351 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:27.390922 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:27.390945 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:27.393242 (+ 2297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:27.397475 (+ 4233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:27.397481 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:27.397485 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:27.397910 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:27.398275 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:27.398280 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:27.398284 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:27.398417 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:27.398958 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:27.398964 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:27.399151 (+ 187us) client_negotiation.cc:769] Sending connection context
0810 23:31:27.399282 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 23:31:27.399574 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260810 23:31:27.400892 19409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:27.383025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47406 (local address 127.14.175.124:38045)
0810 23:31:27.385394 (+ 2369us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:27.385399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:27.385423 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:31:27.385470 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:27.385474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:27.385546 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:27.385643 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:27.386503 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:27.387722 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:27.389322 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:27.389991 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:27.390140 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:27.393442 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:27.393447 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:27.393450 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:27.393483 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:27.397364 (+ 3881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:27.398027 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:27.398034 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:27.398038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:27.398168 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:27.398519 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:27.398524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:27.398528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:27.398828 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:27.400153 (+ 1325us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:27.400286 (+ 133us) server_negotiation.cc:299] Negotiation successful
0810 23:31:27.400504 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2209,"thread_start_us":1961,"threads_started":1}
Aug 10 23:31:27 dist-test-slave-rmhg krb5kdc[19056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404687, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.124@KRBTEST.COM for kudu/127.14.175.125@KRBTEST.COM
Aug 10 23:31:27 dist-test-slave-rmhg krb5kdc[19056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.126@KRBTEST.COM
I20260810 23:31:27.407207 19185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b773c2118c9e4c15b459f0e53fb00713: Bootstrap starting.
I20260810 23:31:27.408358 19293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:27.394366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59528 (local address 127.14.175.125:36277)
0810 23:31:27.394539 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:27.394543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:27.394558 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:31:27.394604 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:27.394608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:27.394663 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:27.394748 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:27.395499 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:27.396307 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:27.397736 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:27.398384 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:27.398484 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:27.402240 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:27.402248 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:27.402253 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:27.402297 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:27.405197 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:27.405997 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:27.406003 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:27.406007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:27.406176 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:27.406637 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:27.406643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:27.406646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:27.407061 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:27.407236 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:27.407719 (+ 483us) server_negotiation.cc:299] Negotiation successful
0810 23:31:27.407942 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 23:31:27.408363 19408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:27.393992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.125:36277 (local address 127.0.0.1:59528)
0810 23:31:27.394144 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:27.394162 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:27.394267 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:27.394798 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:27.394803 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:27.395200 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:27.395351 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:27.395360 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:27.396441 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:27.396446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:27.397557 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:27.397571 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:27.398746 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:27.398752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:27.398920 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:27.399407 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:27.399423 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:27.402051 (+ 2628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:27.405336 (+ 3285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:27.405341 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:27.405344 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:27.405872 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:27.406323 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:27.406329 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:27.406332 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:27.406509 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:27.407247 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:27.407253 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:27.407438 (+ 185us) client_negotiation.cc:769] Sending connection context
0810 23:31:27.407591 (+ 153us) client_negotiation.cc:240] Negotiation successful
0810 23:31:27.407882 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 23:31:27.409636 19400 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.175.124" port: 38045 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:31:27.411556 19185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b773c2118c9e4c15b459f0e53fb00713: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:27.411635 19407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:27.393691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43070 (local address 127.14.175.126:34059)
0810 23:31:27.394257 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:27.394262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:27.394279 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:31:27.394326 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:27.394331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:27.394415 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:27.394506 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:27.396036 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:27.397057 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:27.398780 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:27.399400 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:27.399504 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:27.404956 (+ 5452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:27.404964 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:27.404969 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:27.405010 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:27.408261 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:27.409083 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:27.409089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:27.409092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:27.409303 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:27.409766 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:27.409771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:27.409775 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:27.410140 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:27.410435 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:27.411247 (+ 812us) server_negotiation.cc:299] Negotiation successful
0810 23:31:27.411415 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446}
I20260810 23:31:27.412904 19185 log.cc:824] T 00000000000000000000000000000000 P b773c2118c9e4c15b459f0e53fb00713: Log is configured to *not* fsync() on all Append() calls
I20260810 23:31:27.417820 19185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b773c2118c9e4c15b459f0e53fb00713: No bootstrap required, opened a new log
Aug 10 23:31:27 dist-test-slave-rmhg krb5kdc[19056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404687, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.124@KRBTEST.COM for kudu/127.14.175.124@KRBTEST.COM
Aug 10 23:31:27 dist-test-slave-rmhg krb5kdc[19056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.175.125@KRBTEST.COM
I20260810 23:31:27.427701 19185 raft_consensus.cc:348] T 00000000000000000000000000000000 P b773c2118c9e4c15b459f0e53fb00713 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b773c2118c9e4c15b459f0e53fb00713" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34059 } } peers { permanent_uuid: "5860c14d4d9b4aab9ba7c593cd24990a" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 36277 } } peers { permanent_uuid: "2186c103cb5e4549923d4eb7b21340c1" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 38045 } }
I20260810 23:31:27.428022 19408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:27.410510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:38045 (local address 127.0.0.1:47410)
0810 23:31:27.410697 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:27.410716 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:27.410802 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:27.414120 (+ 3318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:27.414126 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:27.414500 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:27.414647 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:27.414656 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:27.415759 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:27.415763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:27.416872 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:27.416887 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:27.417862 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:27.417869 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:27.418105 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:27.418753 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:27.418775 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:27.421433 (+ 2658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:27.424788 (+ 3355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:27.424795 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:27.424799 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:27.425590 (+ 791us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:27.426028 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:27.426033 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:27.426036 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:27.426215 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:27.426945 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:27.426950 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:27.427126 (+ 176us) client_negotiation.cc:769] Sending connection context
0810 23:31:27.427314 (+ 188us) client_negotiation.cc:240] Negotiation successful
0810 23:31:27.427643 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260810 23:31:27.428149 19185 raft_consensus.cc:729] T 00000000000000000000000000000000 P b773c2118c9e4c15b459f0e53fb00713 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b773c2118c9e4c15b459f0e53fb00713, State: Initialized, Role: FOLLOWER
I20260810 23:31:27.428584 19409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:27.410821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47410 (local address 127.14.175.124:38045)
0810 23:31:27.413302 (+ 2481us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:27.413308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:27.413326 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:31:27.413374 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:27.413378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:27.413465 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:27.413569 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:27.414783 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:27.415620 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:27.417033 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:27.417689 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:27.417826 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:27.421654 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:27.421660 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:27.421665 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:27.421714 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:27.424631 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:27.425735 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:27.425741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:27.425744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:27.425899 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:27.426339 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:27.426345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:27.426348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:27.426807 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:27.426960 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:27.427446 (+ 486us) server_negotiation.cc:299] Negotiation successful
0810 23:31:27.427850 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":116,"server-negotiator.queue_time_us":2361}
I20260810 23:31:27.428691 19185 consensus_queue.cc:260] T 00000000000000000000000000000000 P b773c2118c9e4c15b459f0e53fb00713 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b773c2118c9e4c15b459f0e53fb00713" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34059 } } peers { permanent_uuid: "5860c14d4d9b4aab9ba7c593cd24990a" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 36277 } } peers { permanent_uuid: "2186c103cb5e4549923d4eb7b21340c1" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 38045 } }
I20260810 23:31:27.430245 19293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:27.413521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59530 (local address 127.14.175.125:36277)
0810 23:31:27.413698 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:27.413703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:27.413720 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:31:27.413768 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:27.413773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:27.413838 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:27.413935 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:27.415412 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:27.416507 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:27.417923 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:27.418527 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:27.418623 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:27.423712 (+ 5089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:27.423720 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:27.423726 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:27.423771 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:27.426886 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:27.427792 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:27.427798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:27.427801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:27.427976 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:27.428471 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:27.428476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:27.428479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:27.428835 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:27.429091 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:27.429688 (+ 597us) server_negotiation.cc:299] Negotiation successful
0810 23:31:27.429918 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 23:31:27.433961 19411 sys_catalog.cc:455] T 00000000000000000000000000000000 P b773c2118c9e4c15b459f0e53fb00713 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b773c2118c9e4c15b459f0e53fb00713" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34059 } } peers { permanent_uuid: "5860c14d4d9b4aab9ba7c593cd24990a" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 36277 } } peers { permanent_uuid: "2186c103cb5e4549923d4eb7b21340c1" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 38045 } } }
I20260810 23:31:27.434653 19411 sys_catalog.cc:458] T 00000000000000000000000000000000 P b773c2118c9e4c15b459f0e53fb00713 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:31:27.434978 19400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2186c103cb5e4549923d4eb7b21340c1: Bootstrap starting.
I20260810 23:31:27.435343 19185 sys_catalog.cc:564] T 00000000000000000000000000000000 P b773c2118c9e4c15b459f0e53fb00713 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:31:27.435678 15037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:31:27.438350 19140 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b773c2118c9e4c15b459f0e53fb00713"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43070
I20260810 23:31:27.438683 19140 raft_consensus.cc:482] T 00000000000000000000000000000000 P b773c2118c9e4c15b459f0e53fb00713 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 23:31:27.438884 19140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b773c2118c9e4c15b459f0e53fb00713 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:27.439244 19400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2186c103cb5e4549923d4eb7b21340c1: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:27.440701 19400 log.cc:824] T 00000000000000000000000000000000 P 2186c103cb5e4549923d4eb7b21340c1: Log is configured to *not* fsync() on all Append() calls
I20260810 23:31:27.441533 19140 raft_consensus.cc:504] T 00000000000000000000000000000000 P b773c2118c9e4c15b459f0e53fb00713 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b773c2118c9e4c15b459f0e53fb00713" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34059 } } peers { permanent_uuid: "5860c14d4d9b4aab9ba7c593cd24990a" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 36277 } } peers { permanent_uuid: "2186c103cb5e4549923d4eb7b21340c1" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 38045 } }
I20260810 23:31:27.446491 19400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2186c103cb5e4549923d4eb7b21340c1: No bootstrap required, opened a new log
I20260810 23:31:27.454679 19140 leader_election.cc:288] T 00000000000000000000000000000000 P b773c2118c9e4c15b459f0e53fb00713 [CANDIDATE]: Term 1 election: Requested vote from peers 5860c14d4d9b4aab9ba7c593cd24990a (127.14.175.125:36277), 2186c103cb5e4549923d4eb7b21340c1 (127.14.175.124:38045)
I20260810 23:31:27.457594 19400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2186c103cb5e4549923d4eb7b21340c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b773c2118c9e4c15b459f0e53fb00713" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34059 } } peers { permanent_uuid: "5860c14d4d9b4aab9ba7c593cd24990a" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 36277 } } peers { permanent_uuid: "2186c103cb5e4549923d4eb7b21340c1" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 38045 } }
I20260810 23:31:27.458112 19400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2186c103cb5e4549923d4eb7b21340c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2186c103cb5e4549923d4eb7b21340c1, State: Initialized, Role: FOLLOWER
I20260810 23:31:27.458715 19400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2186c103cb5e4549923d4eb7b21340c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b773c2118c9e4c15b459f0e53fb00713" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34059 } } peers { permanent_uuid: "5860c14d4d9b4aab9ba7c593cd24990a" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 36277 } } peers { permanent_uuid: "2186c103cb5e4549923d4eb7b21340c1" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 38045 } }
I20260810 23:31:27.448603 19077 master_runner.cc:186] Error getting master registration for 127.14.175.125:36277: OK, instance_id { permanent_uuid: "5860c14d4d9b4aab9ba7c593cd24990a" instance_seqno: 1786404686802022 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:31:27.460839 19077 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:31:27.465881 19436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2186c103cb5e4549923d4eb7b21340c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b773c2118c9e4c15b459f0e53fb00713" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34059 } } peers { permanent_uuid: "5860c14d4d9b4aab9ba7c593cd24990a" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 36277 } } peers { permanent_uuid: "2186c103cb5e4549923d4eb7b21340c1" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 38045 } } }
I20260810 23:31:27.466360 19436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2186c103cb5e4549923d4eb7b21340c1 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:31:27.468557 19400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2186c103cb5e4549923d4eb7b21340c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:31:27.468636 19362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b773c2118c9e4c15b459f0e53fb00713" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2186c103cb5e4549923d4eb7b21340c1"
I20260810 23:31:27.469345 19362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2186c103cb5e4549923d4eb7b21340c1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:27.473392 19362 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2186c103cb5e4549923d4eb7b21340c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b773c2118c9e4c15b459f0e53fb00713 in term 1.
I20260810 23:31:27.474340 19099 leader_election.cc:302] T 00000000000000000000000000000000 P b773c2118c9e4c15b459f0e53fb00713 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2186c103cb5e4549923d4eb7b21340c1, b773c2118c9e4c15b459f0e53fb00713; no voters:
I20260810 23:31:27.474874 19411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b773c2118c9e4c15b459f0e53fb00713 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:31:27.475571 19411 raft_consensus.cc:686] T 00000000000000000000000000000000 P b773c2118c9e4c15b459f0e53fb00713 [term 1 LEADER]: Becoming Leader. State: Replica: b773c2118c9e4c15b459f0e53fb00713, State: Running, Role: LEADER
I20260810 23:31:27.476198 19411 consensus_queue.cc:237] T 00000000000000000000000000000000 P b773c2118c9e4c15b459f0e53fb00713 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b773c2118c9e4c15b459f0e53fb00713" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34059 } } peers { permanent_uuid: "5860c14d4d9b4aab9ba7c593cd24990a" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 36277 } } peers { permanent_uuid: "2186c103cb5e4549923d4eb7b21340c1" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 38045 } }
W20260810 23:31:27.481467 19107 tablet.cc:2369] T 00000000000000000000000000000000 P b773c2118c9e4c15b459f0e53fb00713: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:31:27.482558 19430 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b773c2118c9e4c15b459f0e53fb00713: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:31:27.482697 19430 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:31:27.493880 19445 sys_catalog.cc:455] T 00000000000000000000000000000000 P b773c2118c9e4c15b459f0e53fb00713 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b773c2118c9e4c15b459f0e53fb00713. Latest consensus state: current_term: 1 leader_uuid: "b773c2118c9e4c15b459f0e53fb00713" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b773c2118c9e4c15b459f0e53fb00713" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34059 } } peers { permanent_uuid: "5860c14d4d9b4aab9ba7c593cd24990a" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 36277 } } peers { permanent_uuid: "2186c103cb5e4549923d4eb7b21340c1" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 38045 } } }
I20260810 23:31:27.494371 19445 sys_catalog.cc:458] T 00000000000000000000000000000000 P b773c2118c9e4c15b459f0e53fb00713 [sys.catalog]: This master's current role is: LEADER
I20260810 23:31:27.498306 19457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:31:27.500845 19457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:31:27.522017 19407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:27.478061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43078 (local address 127.14.175.126:34059)
0810 23:31:27.478540 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:27.478550 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:27.491028 (+ 12478us) server_negotiation.cc:407] Connection header received
0810 23:31:27.491576 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:27.491583 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:27.491692 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:27.491775 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:27.495083 (+ 3308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:27.496530 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:27.506874 (+ 10344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:27.507590 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:27.507726 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:27.511892 (+ 4166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:27.511898 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:27.511901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:27.511973 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:27.517026 (+ 5053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:27.518819 (+ 1793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:27.518825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:27.518828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:27.518993 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:27.519485 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:27.519491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:27.519495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:27.519923 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:27.520233 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:27.521342 (+ 1109us) server_negotiation.cc:299] Negotiation successful
0810 23:31:27.521598 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355}
I20260810 23:31:27.528957 19288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.175.124" port: 38045 } attempt: 5
I20260810 23:31:27.529482 19409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:27.478340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47426 (local address 127.14.175.124:38045)
0810 23:31:27.478532 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:27.478539 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:27.489920 (+ 11381us) server_negotiation.cc:407] Connection header received
0810 23:31:27.491579 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:27.491587 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:27.491703 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:27.491783 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:27.497545 (+ 5762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:27.498629 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:27.507514 (+ 8885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:27.508276 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:27.508384 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:27.511884 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:27.511892 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:27.511896 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:27.511944 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:27.516496 (+ 4552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:27.517518 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:27.517523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:27.517528 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:27.517717 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:27.518278 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:27.518284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:27.518287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:27.518991 (+ 704us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:27.525460 (+ 6469us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:27.526800 (+ 1340us) server_negotiation.cc:299] Negotiation successful
0810 23:31:27.527897 (+ 1097us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:31:27.531244 19362 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2186c103cb5e4549923d4eb7b21340c1 [term 1 FOLLOWER]: Refusing update from remote peer b773c2118c9e4c15b459f0e53fb00713: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:31:27.532311 19445 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b773c2118c9e4c15b459f0e53fb00713 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2186c103cb5e4549923d4eb7b21340c1" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 38045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Aug 10 23:31:27 dist-test-slave-rmhg krb5kdc[19056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404686, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.175.125@KRBTEST.COM for kudu/127.14.175.124@KRBTEST.COM
I20260810 23:31:27.551033 19406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:27.530915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47432 (local address 127.14.175.124:38045)
0810 23:31:27.531121 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:27.531128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:27.531495 (+ 367us) server_negotiation.cc:407] Connection header received
0810 23:31:27.531550 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:27.531555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:27.531631 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:27.531719 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:27.532870 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:27.533952 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:27.535680 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:27.536346 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:27.536490 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:27.540768 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:27.540776 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:27.540781 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:27.540826 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:27.544632 (+ 3806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:27.545673 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:27.545679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:27.545683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:27.545871 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:27.546624 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:27.546629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:27.546632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:27.547649 (+ 1017us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:27.548463 (+ 814us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:27.549207 (+ 744us) server_negotiation.cc:299] Negotiation successful
0810 23:31:27.550244 (+ 1037us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 23:31:27.551293 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:27.530202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.175.124:38045 (local address 127.0.0.1:47432)
0810 23:31:27.531243 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:27.531265 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:27.531390 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:27.531917 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:27.531922 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:27.532481 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:27.532694 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:27.532706 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:27.534133 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:27.534139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:27.535503 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:27.535519 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:27.536705 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:27.536712 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:27.536950 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:27.537638 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:27.537662 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:27.540494 (+ 2832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:27.544884 (+ 4390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:27.544889 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:27.544893 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:27.545479 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:27.546028 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:27.546033 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:27.546036 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:27.546418 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:27.547872 (+ 1454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:27.547907 (+ 35us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:27.548397 (+ 490us) client_negotiation.cc:769] Sending connection context
0810 23:31:27.549137 (+ 740us) client_negotiation.cc:240] Negotiation successful
0810 23:31:27.550932 (+ 1795us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":908}
I20260810 23:31:27.552631 19436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2186c103cb5e4549923d4eb7b21340c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b773c2118c9e4c15b459f0e53fb00713. Latest consensus state: current_term: 1 leader_uuid: "b773c2118c9e4c15b459f0e53fb00713" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b773c2118c9e4c15b459f0e53fb00713" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34059 } } peers { permanent_uuid: "5860c14d4d9b4aab9ba7c593cd24990a" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 36277 } } peers { permanent_uuid: "2186c103cb5e4549923d4eb7b21340c1" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 38045 } } }
I20260810 23:31:27.552940 19436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2186c103cb5e4549923d4eb7b21340c1 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:31:27.557435 19293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:27.490631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59538 (local address 127.14.175.125:36277)
0810 23:31:27.490834 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:27.490841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:27.520347 (+ 29506us) server_negotiation.cc:407] Connection header received
0810 23:31:27.520427 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:27.520433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:27.520529 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:27.520631 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:27.522180 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:27.523257 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:27.525865 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:27.526497 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:27.526614 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:27.546998 (+ 20384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:27.547006 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:27.547011 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:27.547052 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:27.550782 (+ 3730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:27.552510 (+ 1728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:27.552516 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:27.552519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:27.552698 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:27.553240 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:27.553245 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:27.553248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:27.553650 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:27.553905 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:27.555584 (+ 1679us) server_negotiation.cc:299] Negotiation successful
0810 23:31:27.555853 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 23:31:27.559278 19436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2186c103cb5e4549923d4eb7b21340c1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b773c2118c9e4c15b459f0e53fb00713" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b773c2118c9e4c15b459f0e53fb00713" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34059 } } peers { permanent_uuid: "5860c14d4d9b4aab9ba7c593cd24990a" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 36277 } } peers { permanent_uuid: "2186c103cb5e4549923d4eb7b21340c1" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 38045 } } }
I20260810 23:31:27.559568 19436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2186c103cb5e4549923d4eb7b21340c1 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:31:27.558760 19461 mvcc.cc:205] Tried to move back new op lower bound from 7317113600112422912 to 7317113599908515840. Current Snapshot: MvccSnapshot[applied={T|T < 7317113600112422912}]
I20260810 23:31:27.561268 19445 sys_catalog.cc:455] T 00000000000000000000000000000000 P b773c2118c9e4c15b459f0e53fb00713 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b773c2118c9e4c15b459f0e53fb00713" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b773c2118c9e4c15b459f0e53fb00713" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34059 } } peers { permanent_uuid: "5860c14d4d9b4aab9ba7c593cd24990a" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 36277 } } peers { permanent_uuid: "2186c103cb5e4549923d4eb7b21340c1" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 38045 } } }
I20260810 23:31:27.561554 19445 sys_catalog.cc:458] T 00000000000000000000000000000000 P b773c2118c9e4c15b459f0e53fb00713 [sys.catalog]: This master's current role is: LEADER
I20260810 23:31:27.564997 19288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5860c14d4d9b4aab9ba7c593cd24990a: Bootstrap starting.
W20260810 23:31:27.565224 19474 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2186c103cb5e4549923d4eb7b21340c1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:31:27.565367 19474 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:31:27.574118 19457 catalog_manager.cc:1347] Generated new cluster ID: 0073329275554135828e9be52d6aa9d7
I20260810 23:31:27.574263 19457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:31:27.573329 19299 master_runner.cc:186] Error getting master registration for 127.14.175.125:36277: OK, instance_id { permanent_uuid: "5860c14d4d9b4aab9ba7c593cd24990a" instance_seqno: 1786404686802022 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:31:27.585582 19299 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:31:27.587610 19288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5860c14d4d9b4aab9ba7c593cd24990a: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:27.591353 19288 log.cc:824] T 00000000000000000000000000000000 P 5860c14d4d9b4aab9ba7c593cd24990a: Log is configured to *not* fsync() on all Append() calls
I20260810 23:31:27.596884 19288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5860c14d4d9b4aab9ba7c593cd24990a: No bootstrap required, opened a new log
I20260810 23:31:27.615916 19457 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:31:27.616727 19288 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5860c14d4d9b4aab9ba7c593cd24990a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b773c2118c9e4c15b459f0e53fb00713" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34059 } } peers { permanent_uuid: "5860c14d4d9b4aab9ba7c593cd24990a" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 36277 } } peers { permanent_uuid: "2186c103cb5e4549923d4eb7b21340c1" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 38045 } }
I20260810 23:31:27.617303 19288 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5860c14d4d9b4aab9ba7c593cd24990a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5860c14d4d9b4aab9ba7c593cd24990a, State: Initialized, Role: FOLLOWER
I20260810 23:31:27.617920 19288 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5860c14d4d9b4aab9ba7c593cd24990a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b773c2118c9e4c15b459f0e53fb00713" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34059 } } peers { permanent_uuid: "5860c14d4d9b4aab9ba7c593cd24990a" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 36277 } } peers { permanent_uuid: "2186c103cb5e4549923d4eb7b21340c1" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 38045 } }
I20260810 23:31:27.619583 19457 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:31:27.626157 19479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5860c14d4d9b4aab9ba7c593cd24990a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b773c2118c9e4c15b459f0e53fb00713" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34059 } } peers { permanent_uuid: "5860c14d4d9b4aab9ba7c593cd24990a" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 36277 } } peers { permanent_uuid: "2186c103cb5e4549923d4eb7b21340c1" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 38045 } } }
I20260810 23:31:27.628594 19288 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5860c14d4d9b4aab9ba7c593cd24990a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:31:27.628675 19251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b773c2118c9e4c15b459f0e53fb00713" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5860c14d4d9b4aab9ba7c593cd24990a"
I20260810 23:31:27.629033 19250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5860c14d4d9b4aab9ba7c593cd24990a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:27.630141 19479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5860c14d4d9b4aab9ba7c593cd24990a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:31:27.632848 19250 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5860c14d4d9b4aab9ba7c593cd24990a [term 1 FOLLOWER]: Refusing update from remote peer b773c2118c9e4c15b459f0e53fb00713: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:31:27.636395 19251 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5860c14d4d9b4aab9ba7c593cd24990a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b773c2118c9e4c15b459f0e53fb00713 in term 1.
I20260810 23:31:27.639052 19411 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b773c2118c9e4c15b459f0e53fb00713 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5860c14d4d9b4aab9ba7c593cd24990a" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 36277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:31:27.641225 19457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b773c2118c9e4c15b459f0e53fb00713: Generated new TSK 0
I20260810 23:31:27.642431 19457 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:31:27.706476 19493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5860c14d4d9b4aab9ba7c593cd24990a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b773c2118c9e4c15b459f0e53fb00713" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b773c2118c9e4c15b459f0e53fb00713" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34059 } } peers { permanent_uuid: "5860c14d4d9b4aab9ba7c593cd24990a" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 36277 } } peers { permanent_uuid: "2186c103cb5e4549923d4eb7b21340c1" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 38045 } } }
I20260810 23:31:27.707020 19493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5860c14d4d9b4aab9ba7c593cd24990a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:31:27.706456 19479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5860c14d4d9b4aab9ba7c593cd24990a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b773c2118c9e4c15b459f0e53fb00713. Latest consensus state: current_term: 1 leader_uuid: "b773c2118c9e4c15b459f0e53fb00713" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b773c2118c9e4c15b459f0e53fb00713" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34059 } } peers { permanent_uuid: "5860c14d4d9b4aab9ba7c593cd24990a" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 36277 } } peers { permanent_uuid: "2186c103cb5e4549923d4eb7b21340c1" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 38045 } } }
I20260810 23:31:27.707683 19479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5860c14d4d9b4aab9ba7c593cd24990a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:31:27.749598 19506 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260810 23:31:27.774557 19406 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:31:27.764169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47434 (local address 127.14.175.124:38045)
0810 23:31:27.764384 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:27.764390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:27.771089 (+ 6699us) server_negotiation.cc:407] Connection header received
0810 23:31:27.772705 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:27.772713 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:27.772808 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:27.772889 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:27.774251 (+ 1362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47434: BlockingRecv error: recv got EOF from 127.0.0.1:47434 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260810 23:31:27.778937 19407 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:31:27.763520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43080 (local address 127.14.175.126:34059)
0810 23:31:27.763771 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:27.763778 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:27.775779 (+ 12001us) server_negotiation.cc:407] Connection header received
0810 23:31:27.776517 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:27.776524 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:27.776628 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:27.776719 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:27.778659 (+ 1940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43080: BlockingRecv error: recv got EOF from 127.0.0.1:43080 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260810 23:31:27.779246 19293 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:31:27.771804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59546 (local address 127.14.175.125:36277)
0810 23:31:27.772083 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:27.772090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:27.776222 (+ 4132us) server_negotiation.cc:407] Connection header received
0810 23:31:27.777068 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:27.777074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:27.777205 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:27.777282 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:27.778986 (+ 1704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59546: BlockingRecv error: recv got EOF from 127.0.0.1:59546 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260810 23:31:27.791296 19293 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:31:27.787157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59556 (local address 127.14.175.125:36277)
0810 23:31:27.787371 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:27.787378 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:27.789495 (+ 2117us) server_negotiation.cc:407] Connection header received
0810 23:31:27.789780 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:27.789787 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:27.789928 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:27.790059 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:27.790995 (+ 936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59556: BlockingRecv error: recv got EOF from 127.0.0.1:59556 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260810 23:31:27.807276 19293 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:31:27.803337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59566 (local address 127.14.175.125:36277)
0810 23:31:27.803588 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:27.803596 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:27.805742 (+ 2146us) server_negotiation.cc:407] Connection header received
0810 23:31:27.805940 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:27.805944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:27.806022 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:27.806114 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:27.807076 (+ 962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59566: BlockingRecv error: recv got EOF from 127.0.0.1:59566 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260810 23:31:27.847244 19293 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:31:27.842972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59576 (local address 127.14.175.125:36277)
0810 23:31:27.843181 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:27.843189 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:27.845792 (+ 2603us) server_negotiation.cc:407] Connection header received
0810 23:31:27.845995 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:27.845999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:27.846093 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:27.846187 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:27.846990 (+ 803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59576: BlockingRecv error: recv got EOF from 127.0.0.1:59576 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260810 23:31:27.856559 19293 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:31:27.854487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59584 (local address 127.14.175.125:36277)
0810 23:31:27.854681 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:27.854689 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:27.855189 (+ 500us) server_negotiation.cc:407] Connection header received
0810 23:31:27.855380 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:27.855385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:27.855458 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:27.855543 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:27.856310 (+ 767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59584: BlockingRecv error: recv got EOF from 127.0.0.1:59584 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260810 23:31:27.869969 19293 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:31:27.867644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59588 (local address 127.14.175.125:36277)
0810 23:31:27.867870 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:27.867883 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:27.868408 (+ 525us) server_negotiation.cc:407] Connection header received
0810 23:31:27.868608 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:27.868612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:27.868695 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:27.868790 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:27.869730 (+ 940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59588: BlockingRecv error: recv got EOF from 127.0.0.1:59588 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260810 23:31:27.891218 19293 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:31:27.889037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59590 (local address 127.14.175.125:36277)
0810 23:31:27.889286 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:27.889294 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:27.889827 (+ 533us) server_negotiation.cc:407] Connection header received
0810 23:31:27.890021 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:27.890025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:27.890100 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:27.890187 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:27.890978 (+ 791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59590: BlockingRecv error: recv got EOF from 127.0.0.1:59590 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260810 23:31:27.928486 19293 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:31:27.926305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59596 (local address 127.14.175.125:36277)
0810 23:31:27.926517 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:27.926524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:27.927089 (+ 565us) server_negotiation.cc:407] Connection header received
0810 23:31:27.927291 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:27.927295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:27.927370 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:27.927460 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:27.928223 (+ 763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59596: BlockingRecv error: recv got EOF from 127.0.0.1:59596 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260810 23:31:27.998189 19293 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:31:27.995886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59612 (local address 127.14.175.125:36277)
0810 23:31:27.996150 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:27.996158 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:27.996702 (+ 544us) server_negotiation.cc:407] Connection header received
0810 23:31:27.996900 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:27.996904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:27.996988 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:27.997077 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:27.997949 (+ 872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59612: BlockingRecv error: recv got EOF from 127.0.0.1:59612 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 23:31:28.076020 19411 sys_catalog.cc:455] T 00000000000000000000000000000000 P b773c2118c9e4c15b459f0e53fb00713 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b773c2118c9e4c15b459f0e53fb00713" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b773c2118c9e4c15b459f0e53fb00713" member_type: VOTER last_known_addr { host: "127.14.175.126" port: 34059 } } peers { permanent_uuid: "5860c14d4d9b4aab9ba7c593cd24990a" member_type: VOTER last_known_addr { host: "127.14.175.125" port: 36277 } } peers { permanent_uuid: "2186c103cb5e4549923d4eb7b21340c1" member_type: VOTER last_known_addr { host: "127.14.175.124" port: 38045 } } }
I20260810 23:31:28.076454 19411 sys_catalog.cc:458] T 00000000000000000000000000000000 P b773c2118c9e4c15b459f0e53fb00713 [sys.catalog]: This master's current role is: LEADER
W20260810 23:31:28.131625 19293 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:31:28.129404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59614 (local address 127.14.175.125:36277)
0810 23:31:28.129625 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:28.129631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:28.130155 (+ 524us) server_negotiation.cc:407] Connection header received
0810 23:31:28.130361 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:28.130366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:28.130440 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:28.130527 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:28.131359 (+ 832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59614: BlockingRecv error: recv got EOF from 127.0.0.1:59614 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260810 23:31:28.396602 19293 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:31:28.394771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59616 (local address 127.14.175.125:36277)
0810 23:31:28.394994 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:28.395001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:28.395224 (+ 223us) server_negotiation.cc:407] Connection header received
0810 23:31:28.395442 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:28.395447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:28.395521 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:28.395607 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:28.396382 (+ 775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59616: BlockingRecv error: recv got EOF from 127.0.0.1:59616 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 23:31:28.567435 19474 catalog_manager.cc:1259] Loaded cluster ID: 0073329275554135828e9be52d6aa9d7
I20260810 23:31:28.567559 19474 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2186c103cb5e4549923d4eb7b21340c1: loading cluster ID for follower catalog manager: success
I20260810 23:31:28.570308 19474 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2186c103cb5e4549923d4eb7b21340c1: acquiring CA information for follower catalog manager: success
I20260810 23:31:28.572180 19474 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2186c103cb5e4549923d4eb7b21340c1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:31:28.754155 19506 catalog_manager.cc:1259] Loaded cluster ID: 0073329275554135828e9be52d6aa9d7
I20260810 23:31:28.754317 19506 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5860c14d4d9b4aab9ba7c593cd24990a: loading cluster ID for follower catalog manager: success
I20260810 23:31:28.757390 19506 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5860c14d4d9b4aab9ba7c593cd24990a: acquiring CA information for follower catalog manager: success
I20260810 23:31:28.759397 19506 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5860c14d4d9b4aab9ba7c593cd24990a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:31:28.921662 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:28.915147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59630 (local address 127.14.175.125:36277)
0810 23:31:28.916032 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:28.916037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:28.916053 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:31:28.916117 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:28.916121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:28.916285 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:28.916401 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:31:28.916919 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:28.917820 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:28.919202 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:28.919685 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:28.919805 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:28.920347 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:31:28.920987 (+ 640us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:31:28.921134 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:28.921276 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 23:31:28.921363 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":763,"thread_start_us":570,"threads_started":1}
I20260810 23:31:28.934679 19610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:28.928012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47448 (local address 127.14.175.124:38045)
0810 23:31:28.928774 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:28.928780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:28.928800 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:31:28.928914 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:28.928920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:28.929124 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:28.929331 (+ 207us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:31:28.929668 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:28.930790 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:28.932064 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:28.932665 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:28.932780 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:28.933373 (+ 593us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:31:28.934103 (+ 730us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:31:28.934254 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:28.934303 (+ 49us) server_negotiation.cc:299] Negotiation successful
0810 23:31:28.934388 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":494,"threads_started":1}
I20260810 23:31:28.956207 19610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:28.941717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47462 (local address 127.14.175.124:38045)
0810 23:31:28.945490 (+ 3773us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:28.945496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:28.948577 (+ 3081us) server_negotiation.cc:407] Connection header received
0810 23:31:28.948782 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:28.948788 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:28.948902 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:28.948993 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:31:28.949776 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:28.950992 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:28.953404 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:28.954111 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:28.954212 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:28.954922 (+ 710us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:31:28.955097 (+ 175us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:31:28.955168 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:28.955763 (+ 595us) server_negotiation.cc:299] Negotiation successful
0810 23:31:28.955863 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3630}
I20260810 23:31:28.963500 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:28.941364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43082 (local address 127.14.175.126:34059)
0810 23:31:28.956672 (+ 15308us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:28.956679 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:28.956702 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:31:28.956824 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:28.956830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:28.957049 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:28.957267 (+ 218us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:31:28.958207 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:28.959139 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:28.960533 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:28.961215 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:28.961340 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:28.961934 (+ 594us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:31:28.962453 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:31:28.963021 (+ 568us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:28.963085 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 23:31:28.963177 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15164,"thread_start_us":16670,"threads_started":1}
I20260810 23:31:28.963783 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:28.949141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59636 (local address 127.14.175.125:36277)
0810 23:31:28.949377 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:28.949383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:28.958001 (+ 8618us) server_negotiation.cc:407] Connection header received
0810 23:31:28.958304 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:28.958310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:28.958398 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:28.958489 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:31:28.959042 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:28.960218 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:28.962026 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:28.962451 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:28.962561 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:28.963007 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:31:28.963131 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:31:28.963203 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:28.963360 (+ 157us) server_negotiation.cc:299] Negotiation successful
0810 23:31:28.963443 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 23:31:28.969511 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 19077
I20260810 23:31:28.970530 19176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:31:29.079834 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 19077
W20260810 23:31:29.105744 19219 connection.cc:535] server connection from 127.0.0.1:59512 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 23:31:29.105950 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 19194
I20260810 23:31:29.107141 19283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:31:29.213099 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 19194
I20260810 23:31:29.241849 15037 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu pid 19299
I20260810 23:31:29.243093 19394 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:31:29.349923 15037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwnDIbQ/build/asan/bin/kudu with pid 19299
2026-08-10T23:31:29Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5206 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10461 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (389774 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST